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Overview 


Introduction 


This report presents 30-month impact results from a random assignment evaluation of the Young 
Adult Internship Program (Y AIP), a subsidized employment program for young people in New York 
City who have become disconnected from school and work. Operated by various provider agencies, 
YAIP offers disconnected young people between the ages of 16 and 24 a temporary paid internship, 
as well as various support services. 


The YAIP evaluation was part of the larger Subsidized and Transitional Employment Demonstration, 
sponsored by the Administration for Children and Families in the U.S. Department of Health and 
Human Services. From July 2013 to March 2014, program staff assigned nearly 2,700 young people 
at random to either a program group, which was offered YAIP services, or to a control group, which 
was not offered those services. The YAIP evaluation measured outcomes for both groups over time 
to assess whether YAIP services led to better outcomes for the program group compared with those 
of the control group. 


This report, the second of two from the YAIP evaluation, examines whether the program improved 
young people’s outcomes 30 months after study enrollment. An analysis of youth outcomes indicates 
that young people in the program and control groups were faring similarly after 30 months, with pro- 
gram group members slightly more likely to report employment on a survey administered roughly 30 
months after random assignment. However, administrative data did not show employment effects, 
suggesting that the program may have increased informal or independent employment. A cost analysis 
found that the program cost $5,431 per participant, which is at the lower end of the spectrum of costs 
of similar programs. 


Primary Research Questions 


This study seeks to answer the following research questions: 
e How was YAIP designed and operated? 


e What impact did YAIP have on employment and earnings, education and training, and well-being 
relative to what would have happened in the absence of the program? Did YAIP appear to be 
more effective for certain subgroups of young people? 


e What were the costs of YAIP’s services? 


Purpose 


For many young people, the time between one’s late teenage years and early twenties encompasses 
several important milestones, including graduating from high school, attending college, entering the 
workforce, and beginning to establish economic independence. However, 12.3 percent of young peo- 
ple in the United States between the ages of 16 and 24 — 4.9 million people in total — are neither in 
school nor working. These “disconnected youth” or “opportunity youth” face serious challenges to 
achieving labor market success and self-sufficiency in adulthood. 


YAIP is intended to help reengage young people who have fallen off track, thereby reducing their risk 
of long-term economic hardship. MDRC conducted a random assignment evaluation of YAIP to de- 
termine whether the program makes a difference in the lives of the young people it serves. 


Key Findings 


As expected, YAIP’s offer of subsidized employment led to large improvements in employment and 
earnings during the first year of follow-up, more than doubling earnings and increasing employment 
by nearly 30 percentage points compared with the control group. Further, findings suggest that the 
program group worked in slightly better jobs than the control group for over a year after most people 
had left the program. However, at the end of the 30-month follow-up period, program group members 
had not successfully made the transition into one of YAIP’s key outcomes — education, employment, 
the military, or training — at a higher rate than they would have in the absence of the program. The 
program group also earned about the same amount of money as the control group. YAIP did increase 
survey-reported employment at the end of the follow-up period, but not employment as measured by 
administrative employment records, indicating that program group members may have been more 
likely to hold jobs in the informal economy or as independent contractors. YAIP did not lead to any 
statistically significant effects in the domains of education or well-being. 


Long-term program effects did not vary much across different populations and locations. Subgroups 
of participants that benefited the most from YAIP’s services in the short term did not retain many 
long-term benefits. During the one-year follow-up period, populations with larger barriers to employ- 
ment at study enrollment (namely, those without a high school credential or those who were discon- 
nected for longer) benefited the most from the program, but those benefits did not last through 30 
months of follow-up. Further, program group members in the YAIP locations that provided the strong- 
est contrast of services relative to their local alternatives, as measured by earnings in the first year of 
follow-up, did not perform better on key outcomes than their control group counterparts 30 months 
after the program began. 


Methods 


The evaluation included an implementation study, an impact study, and a cost analysis. This report 
presents 30-month impact findings and results from the cost analysis. Implementation findings and 
shorter-term impact findings (after 12 months) were presented in a report released in April 2017. 


The implementation study described Y AIP’s design and how the program operated. Main data sources 
for the implementation study included staff interviews, observations, and participation data. 


The impact study used a randomized controlled trial design in which individuals eligible for and in- 
terested in YAIP were randomly assigned to either a program group, which was offered YAIP ser- 
vices, or to a control group, which was not offered those services. The study evaluated impacts on 
employment and earnings, education and training, and well-being. Data sources for the impact study 
included administrative records on wages and postsecondary enrollment, subsidized employment pay- 
roll records, and surveys conducted approximately 4, 12, and 30 months after participants entered the 
study. 


vi 


The cost study assessed the cost of the program per program group member and compared this value 
with similar programs and programs serving a similar population. To determine the cost of the pro- 
gram, the research team examined operating costs, costs of support services, and wages and payroll 
costs recorded in Fiscal Year 2014 — the period when all sample members received the majority of 
their services — using data supplied by YAIP providers and oversight organizations. 


vii 
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Chapter 1 


Introduction 


For many young people, the time between one’s late teenage years and early twenties encom- 
passes several important milestones, including graduating from high school, attending college, 
entering the workforce, and beginning to establish economic independence. For some, however, 
staying engaged in education or employment during the transition to adulthood can be difficult: 
12.3 percent of young people in the United States between the ages of 16 and 24 — 4.9 million 
young people in total — are neither in school nor working. This group is commonly referred to 
as “disconnected youth” or “opportunity youth.” As a result of low levels of educational attain- 
ment and limited work experience, many disconnected youth face serious challenges to achieving 
labor market success and self-sufficiency in adulthood. ! 


New York City’s Young Adult Internship Program (Y AIP) is intended to help reengage 
young people who have fallen off track, thereby reducing their risk of long-term economic hard- 
ship. YAIP offers young people various services, including job-readiness workshops and activi- 
ties; individual support, counseling, and assessments; case management; and follow-up services. 
However, the central program component is a 10- to 12-week paid internship. The Mayor’s Office 
for Economic Opportunity (NYC Opportunity, formerly the New York City Center for Economic 
Opportunity) and the New York City Department of Youth and Community Development 
(DYCD) oversee the program, and community-based provider organizations throughout the city 
deliver it.* 


This report presents final impact and cost study findings through 30 months from a ran- 
dom assignment evaluation of YAIP, which studied the program’s implementation, costs, and 
impacts. The evaluation was funded by the Administration for Children and Families in the U.S. 
Department of Health and Health and Human Services and was conducted by MDRC. In sum, 
although YAIP led to substantial increases in employment and earnings during the program, it 
did not have statistically significant effects on the two primary long-term outcomes assessed: (1) 
earnings in the last year of follow-up and (2) engagement in YAIP’s target activities of employ- 
ment, education, or training at the end of the 30-month follow-up period. The program had long- 
term effects on a few other measures of employment, but no long-term effects on education and 
well-being. Effects did not vary much by participant or location characteristics. 


'Burd-Sharps and Lewis (2017). 


*NYC Opportunity provided initial funding for YAIP, and the New York City Young Men’s Initiative pro- 
vided additional funding to expand the program in 2011. The two groups continue to fund the program today. 


Background 


Many young people in the United States are neither enrolled in school nor participating in the 
labor market. The detachment of these young people from society’s larger structures of school 
and work poses serious costs to their future well-being, their communities, and to the country as 
a whole.’ In recent years, concern about at-risk, out-of-school, and out-of-work young people has 
grown among policymakers, service providers, and other key stakeholders. This concern has gen- 
erated new policies and initiatives to better serve and reconnect this population to education, train- 
ing, and employment. Disconnected youth are a heterogeneous group in terms of the causes of 
their disconnection, educational backgrounds, length of disconnection, and so on; thus, appropri- 
ate service models vary considerably. A number of programs targeted to this population have 
been rigorously evaluated, with some showing positive, statistically significant effects, primarily 
on employment and earnings (for example, Job Corps and National Guard Youth ChalleNGe).4 


YAIP is unique among disconnected youth programs in a few respects: It is a relatively 
simple model focused on work experience, it targets a more job-ready subset of disconnected 
youth, and it operates at large scale.° Thus, learning about the effectiveness of Y AIP will contrib- 
ute to the existing research evidence regarding what works to reengage low-income disconnected 
youth in education and work. 


The Young Adult Internship Program (Y AIP) 


Introduced in 2007, YAIP is a workforce development program designed for young New York 
City residents living in poverty who are neither in school nor working, but who are believed to 
have the potential to benefit from a relatively brief, non-intensive intervention. 


YAIP is a multiphase program that enrolls young adults in cohorts, with a new cohort 
starting every four months. Participants in a particular cohort move through the program to- 
gether. Each community-based Y AIP provider is responsible for enrolling and serving a portion 
of the full cohort, usually about 30 young adults each. The program’s three phases are as follows: 


e Phase 1: The first 2 to 4 weeks of the program (duration varies by provider) 
are referred to as the orientation phase, wherein young people are expected to 
attend daily job-readiness workshops facilitated by program staff at provider 


3Burd-Sharps and Lewis (2017). 

4Treskon (2016); Hossain and Bloom (2015). 

°In the 2015 fiscal year (July 1, 2014, through June 30, 2015), 18 YAIP providers across New York City 
served 1,821 young people. 


offices. Young people are paid minimum wage for 25 hours per week,° and 
workshops are typically five hours per day. The goal of orientation is to pre- 
pare participants for the workplace. 


e Phase 2: During the 10 to 12 weeks of this phase, young people are expected 
to work 20 hours a week in their internship placement and continue to earn 
minimum wage. Their earnings are fully subsidized. Once a week, young peo- 
ple are required to return to the provider offices to attend five-hour educational 
workshops, for which they are also paid minimum wage. 


e Phase 3: The nine months following participants’ completion of their intern- 
ship is the follow-up phase of YAIP. During this time, providers are expected 
to help participants secure and maintain an “outcome placement.” DYCD- 
approved outcome placements include participation in unsubsidized employ- 
ment, education, training, or the military. Providers also offer support services 
during this phase. 


The YAIP Evaluation 


The YAIP evaluation was part of the Subsidized and Transitional Employment Demonstration 
(STED), a national study of subsidized employment programs. Funded by the U.S. Department 
of Health and Human Services, and designed to advance the field’s understanding of subsidized 
employment, STED is studying eight different subsidized employment program models in six 
cities across the country. Each model, including YAIP, was evaluated independently in a random- 
ized controlled trial. 


The evaluation included 12 independent Y AIP providers that delivered the program at 13 
different locations across Brooklyn, the Bronx, Manhattan, and Queens.’ YAIP providers targeted 
individuals between 16 and 24 years of age who were neither in school nor working, among other 


°The minimum wage increased from $7.25 to $8 per hour in New York during the study period. As of this 
report’s publication, the minimum wage in New York City was $13 per hour for employers of 11 or more em- 
ployees and $12 for employers of 10 or fewer employees. 

7At the time of the study, 17 providers were operating YAIP. In a joint decision, NYC Opportunity, DYCD, 
and the research team excluded YAIP providers that also operated programs very similar to YAIP because of the 
high likelihood that young adults assigned to the control group would access those services. In addition, the team 
excluded a small number of providers because NYC Opportunity and DYCD determined that those providers 
would be unable to comply with the study requirements and run the Y AIP program effectively during the study 
period. One provider, Opportunities for a Better Tomorrow, served two Y AIP cohorts in each cycle from differ- 
ent locations. Thus, 12 agencies operated 13 programs in the STED evaluation of YAIP. 


eligibility criteria.° YAIP providers were required to enroll at least 80 percent of participants from 
targeted high-need community districts, characterized by high poverty rates. Figure 1.1 shows a 
map of YAIP provider locations and high-need community districts, as well as the distribution of 
types of provider agencies. 


The YAIP evaluation enrolled a total of 2,678 young people in three consecutive cohorts, 
beginning with the July 2013 cohort, followed by the November 2013 cohort, and concluding 
with the March 2014 cohort.° Y AIP program staff randomly assigned 60 percent of the sample to 
the program group and 40 percent to the control group except when another ratio was needed to 
fill open program slots. Ultimately, 61 percent of study participants were randomly assigned to 
the program group. !° 


e The program group. The 1,638 individuals who were randomly assigned to 
this group were offered YAIP program services, including a paid internship, 
job-readiness training, case management, and follow-up services. 


e The control group. The 1,040 individuals who were randomly assigned to 
this group were not offered YAIP program services but were able to access 
other services that were available in the community including, in some cases, 
other non-Y AIP services offered at YAIP provider agencies. 


The YAIP evaluation has three components: an implementation study, an impact study, 
and a cost study. The goal of the implementation study was to describe the design of the YAIP 
program and how it operates. Findings from the implementation study were presented in the in- 
terim report. The impact study addresses the question of whether Y AIP improves key outcomes 
of interest for disconnected youth. One-year impact results were presented along with implemen- 
tation findings in the interim report.'! The cost study analyzed the price of delivering YAIP ser- 
vices. This report presents analyses of the 30-month impacts of YAIP, as well as results of the 
cost study. 


8y AIP provider staff obtained parental consent for minors participating in the evaluation. 


°Y AIP serves three cohorts of young people each year, enrolling participants in March, July, and November. 
Young people enrolled in a cohort move through the program together. 

10M order to ensure that all open program slots were filled during the evaluation, the research team occa- 
sionally shifted the random assignment ratio to allow for a higher proportion of applicants to be assigned to the 
program group. For this reason, the final proportion of young people assigned to the program group is slightly 
above 60 percent. To adjust for small variations in random assignment ratios and sample size between providers 
and cohorts, the team applied proportional weights to the impact analyses. These weights ensure that estimated 
impacts are not biased by differences across cohorts or providers in the proportion assigned to the program group. 


‘1Skemer, Sherman, Williams, and Cummings (2017). 
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Phase 3: Follow-up Support 
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Internship participants had an average 
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SOURCE: MDRC calculations using data from the YAIP management information system. 


NOTES: Program group sample size: 1,638. Estimates in dotted boxes are calculated among 
internship participants; all other estimates are calculated among all program group members. 
Twelve YAIP providers implemented the program in 13 locations. 


30-Month Impact Analysis Data Sources and Methods 


The research team estimated the program’s effects on key outcomes by measuring them 
approximately 30 months after participants enrolled in the study (unless otherwise specified). 
Outcomes were calculated based on the following data sources: 


e Employment and earnings records. The National Directory of New Hires 
(NDNH) contains data collected by state workforce agencies for jobs covered 
by unemployment insurance. Although this database includes most formal em- 
ployment, there are gaps in unemployment insurance coverage for independ- 
ent contract employment, which are estimated to be 13 percent or higher.” In 
addition to NDNH records, DYCD provided data from its management infor- 
mation system on young people’s subsidized earnings. 


e Survey data. The survey firm Decision Information Resources fielded a sur- 
vey instrument at approximately 30 months after study enrollment. The survey 
was fielded to the full study sample and was completed by 2,031 members of 
the full study sample of 2,678, resulting in a response rate of 76 percent. The 
survey contained questions about employment, education, training, and per- 
sonal and economic well-being. 


e Postsecondary enrollment records. The National Student Clearinghouse 
(NSC) includes data on enrollment in most two- and four-year postsecondary 
institutions in the United States. In 2016, the NSC covered 97 percent of en- 
rollments in United States postsecondary institutions, with the largest gaps in 
coverage among private, for-profit two- and four-year institutions (71 percent 
and 81 percent coverage, respectively). 


The outcomes assessed in this report were selected based on a review of YAIP back- 
ground materials and disconnected youth literature and fall into three domains: employment and 
earnings, education and training, and personal and economic well-being. This report divides im- 
pact findings into two categories: confirmatory and exploratory. Confirmatory findings provide 
conclusive evidence of the program’s effects, while exploratory findings provide suggestive evi- 
dence. Box 1.1 defines and explains these two categories of findings, along with information 
about the methodological considerations and approaches used for selecting confirmatory out- 
comes. 


"Hotz and Scholz (2002). 
‘3Dundar and Shapiro (2016). 


Box 1.1 


Confirmatory and Exploratory Analyses 


This study was designed to provide credible evidence about YAIP’s long-term effects on em- 
ployment and education. To ensure the most rigorous results, the study includes two types of 
analysis: (1) a confirmatory analysis to determine the overall effectiveness of the programs, and 
(2) an exploratory analysis designed to offer additional insight and generate hypotheses for fu- 
ture research. 


Confirmatory analysis uses a high standard of evidence for deciding whether an intervention 
has had its intended effect in order for its findings to be considered conclusive rather than merely 
suggestive. In particular, it is designed to avoid the statistical problem induced by testing multi- 
ple hypotheses at the same time, often referred to as the “multiple comparisons” problem. In 
brief, when many statistical tests are performed simultaneously, the overall probability of a spu- 
rious finding (that is, one due to chance rather than a true program effect) can be substantially 
higher than the reported p-value for each individual test. The confirmatory analysis in this report 
mitigates the multiple comparisons problem by designating two outcomes that best measure 
YAIP’s progress toward its primary long-term program goal of “facilitat[ing] the long-term em- 
ployment and self-sufficiency of youth.” These outcomes were specified before any data analy- 
sis. 


Exploratory analysis looks for suggestive evidence of the program’s impacts on other out- 
comes and on subgroups of interest. Findings from exploratory analyses, which are viewed as 
the best available evidence on potential program effects in secondary areas, can help inform 
policy but should not be taken as definitive. In the exploratory analysis, formal adjustments for 
multiple comparisons are not made when reporting on statistical significance. In this report, the 
language used to describe exploratory results is weaker than the language used for confirmatory 
findings. 


As discussed in the interim report, Y AIP achieved its short-term goal of employing many 
young people who would not otherwise have worked. The confirmatory analysis presented in this 
report assesses whether these short-term improvements translate into longer-term effects on two 
primary outcomes that YAIP was designed to affect: 


e Currently employed, in school, or in vocational training. The research team 
chose this measure for the confirmatory analysis because it reflects current ac- 
tivity in any of YAIP’s key program outcome areas: employment, education, 
vocational training, and military service.'* Program architects designed Y AIP 
to encourage involvement in these activities, with the belief that engagement 


4A lthough military enlistment is not specifically accounted for in this measure, it would be captured by the 
employment measures. 


in any of these activities would improve long-term economic prospects. Thus, 
this measure is a proximal outcome, and it may reflect progress toward YAIP’s 
primary distal outcome of sustained employment with continual growth in 
earnings. This measure is composed of data from the 30-month follow-up sur- 
vey, the NSC, and the NDNH.'® 


e Earnings in the final year of follow-up, which begins in the sixth quarter 
after the quarter of random assignment and ends after the ninth quarter after 
the quarter of random assignment. The research team chose this follow-up pe- 
riod because it reflects a period after which nearly all program group members 
have left subsidized employment and is therefore a clean measure of post-pro- 
gram employment. The confirmatory analysis uses the earnings outcome be- 
cause it reflects several aspects of employment: levels of employment, con- 
sistency of employment, hours worked, and wages paid. This measure is 
calculated based on data from the NDNH. 


In addition to these confirmatory analyses, this report presents exploratory analyses, 
which provide additional insight on the program, but should not be considered definitive. Explor- 
atory analyses include (1) impacts on additional employment and education measures, as well as 
impacts on measures of economic and personal well-being, (2) impacts for subgroups of young 
people, defined by age, length of disconnection, gender, and cohort, and (3) variation in impacts 
by location and a key location-level characteristic. 


Cost Analysis Data Sources and Methods 


The cost study assessed the cost of the program per program group member and com- 
pared this value with similar programs and programs serving a similar population. In order to 
determine the cost of the program, the research team examined operating costs, costs of support- 
ive services, and wages and payroll costs recorded in the 2014 fiscal year, the period when all 
three cohorts received the majority of their services. The cost analysis uses the following data 
sources: 


e NYC Opportunity and the Young Men’s Initiative (YMIJ) cost reports. 
NYC Opportunity and YMI, a public-private partnership created to address 
increasing disparities among black and Latino men between the ages of 16 and 
24, funded the YAIP programs. These reports list the payroll processing fees, 


‘SThis measure was constructed such that engagement on any measure at the time of the survey would indi- 
cate overall engagement. 


workers’ compensation, and payroll taxes for clients who received their fund- 
ing. 


e Provider budget reports and salary information. The research team col- 
lected budget reports and salary information from the individual program 
providers for the 2014 fiscal year (July 2013 through June 2014). The budget 
reports included the funds provided to the contracted providers from NYC Op- 
portunity and YMI, including staff salaries and benefits; costs of overhead al- 
located to the program, which included rent, utilities, equipment, and supplies; 
other direct costs, such as travel, training, and liability insurance; and the costs 
of support services. Salary information captured information on actual salaries 
paid to each YAIP staff member. 


e New York City Department of Youth and Community Development 
(DYCD) cost estimates. DYCD staff oversaw the YAIP program. DYCD es- 
timated how much staff time was spent administering Y AIP and the costs of 
administering the program. 


e Staff time study. A staff time study was administered to all YAIP program 
staff at two points in time: June 1 through June 7, 2014, and August 3 through 
August 9, 2014. YAIP staff recorded how they spent their time during these 
periods. 


e Program participation data. Data from DYCD’s management information 
system captured program group members’ participation in Y AIP and partici- 
pants’ subsidized earnings. 


Sample Member Characteristics 


The diversity of the disconnected youth population is reflected in the YAIP sample. As shown in 
Table 1.1, the YAIP sample was 21 years of age, on average, at the time of study enrollment and 
was divided about evenly between men and women." The vast majority of sample members were 
black, non-Hispanic (58 percent) or Hispanic (36 percent). Sixty-two percent of the sample had 
earned a high school diploma or equivalency certificate and nearly three-fourths of the sample 
had previous work experience. The median length of time since young people in the study were 
last in school, enrolled in a high school equivalency program, or working was nine months. In 
addition to their limited educational and employment histories, 42 percent of sample members 
faced at least one other significant barrier to employment, including limited literacy or math skills, 


‘Table 1.1 is a condensed version of the baseline table displayed in the interim report. Please see the Y AIP 
interim report for a more detailed list of baseline measures. 


Table 1.1 


Characteristics of Sample Members at Baseline 


Characteristic Total 
Average age (years) 20.7 
Male (%) 49.1 
Race (%) 
Hispanic 36.2 
Black, non-Hispanic 58.1 
Other 5.7 


Highest degree achieved* (%) 


No degree 38.3 
GED certificate 7.3 
High school diploma 32.7 
Some postsecondary education, but no postsecondary degree 21.3 
Bachelor's degree or higher 0.4 
Ever employed (%) 71.7 
Median months since last in school, enrolled in GED program, or working 9.1 
Has a child (%) 20.0 
Receives public assistance? (%) 26.3 
Ever arrested (%) 26.0 
Has any of the following barriers to employment (%) 45.7 
Limited literacy or math skills 13.0 
Ever a runaway, homeless, or lived in temporary or emergency housing 5.9 
Ever in foster care 8.9 
Pregnant or has child 21.6 
Has a mental or physical disability 4.3 
Ever convicted of a crime 8.2 
Sample size 2,678 


SOURCES: MDRC calculations based on data from MDRC's random assignment system and the YAIP 
management information system. 


NOTES: GED = General Educational Development. 

aStudents who obtained a high school certificate of completion but not a high school diploma or GED certificate 
are shown as having no degree. 

>This measure includes food stamps (Supplemental Nutrition Assistance Program), family assistance 
(Temporary Assistant for Needy Families), disability (Supplemental Security Income), safety net assistance, 
unemployment compensation, and other unspecified sources of income. 
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housing instability, parenting responsibilities, a disability, or a criminal record. Young people in 
the study sample compare favorably with the broader population of disconnected youth in New 
York City on various socioeconomic indicators, suggesting that YAIP providers successfully tar- 
geted a more job-ready subset of disconnected youth. '” 


Implementation and Early Impacts of YAIP 


The interim report described findings from the implementation study. Overall, the implementa- 
tion of YAIP aligned with the program model and was consistent across providers, allowing for 
a good test of the program’s effects. As shown in Figure 1.1, the program had fairly strong reten- 
tion: More than three-quarters of the program group worked in a DYCD-subsidized internship, 
and, among this group, 86 percent completed their internship.'® YAIP participants generally be- 
lieved they gained important “world of work” skills as a result of the program, though many were 
skeptical that the program adequately prepared them for future employment. Indeed, although 
YAIP was developed as a “light-touch” intervention to help reengage a mostly job-ready subset 
of disconnected youth, many involved with YAIP, including provider staff, supervisors at intern- 
ship sites, and young people themselves, believed that YAIP participants required both a higher 
level of support and a longer intervention to improve their educational and labor market out- 
comes. 


The interim report also assessed one-year impacts on a number of key outcomes to ex- 
amine treatment contrast — or the difference between services received by the two research 
groups — as well as the program’s shorter-term effects. Control group members were free to seek 
out and receive assistance from other programs and organizations in the community, and indeed 
they received many services during the first year of follow-up: Over half (53 percent) reported 
receiving help finding or keeping a job, and nearly half (46 percent) reported receiving advice or 
support from a staff member at an agency. Still, program group members were much more likely 
than control group members to report receiving these services, with 85 percent reporting having 
received help finding or keeping a job, representing a substantial increase in receipt of employ- 
ment-related services for program group members. 


Consistent with prior research on subsidized and transitional jobs, YAIP increased em- 
ployment during the program, but this employment effect dissipated by the end of the one-year 
follow-up period.'® However, program group members had higher earnings than control group 


'7Y ATP sample baseline characteristics were compared with characteristics of disconnected youth in New 
York City, presented in Burd-Sharps and Lewis (2017). 

'8The program group includes all individuals who enrolled in the study and were offered program services, 
including those who never returned to the program after random assignment and those who never completed 
orientation. 


'SRedcross et al. (2016); Glosser, Barden, and Williams (2016); Walter, Navarro, Anderson, and Tso (2017). 
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members throughout the one-year follow-up period. In fact, YAIP more than doubled earnings in 
the first year of follow-up, and the earnings increase was more than double the YAIP subsidy, 
indicating that the program group had higher unsubsidized earnings than the control group during 
that period. Considered together with survey results that showed positive impacts on permanent 
and full-time current employment, these results suggest that program group members found 
slightly better jobs after participating in YAIP. Program and control group members had similar 
outcomes during the first year of follow-up in all other key domains. 


Roadmap to the Report 


Chapter 2 presents 30-month impact findings. Chapter 3 summarizes an analysis of the variation 
in findings across subgroups and provider locations. Chapter 4 presents findings from the cost 
analysis, and Chapter 5 discusses conclusions and policy implications. 
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Chapter 2 


30-Month Impacts 


The interim report presented impacts covering a 12-month follow-up period. This follow-up pe- 
riod included the time when the majority of program group members were engaged in a YAIP 
subsidized internship; thus, the early impacts discussed in that report were not necessarily indic- 
ative of the longer-term effects of the program. This chapter presents Y AIP’s longer-term effects 
on two confirmatory measures — earnings and current engagement in work, school, or training 
— as well as the program’s suggestive effects on other measures of employment, education, and 
well-being. 


In sum, the 30-month analysis found that YAIP did not have statistically significant ef- 
fects on earnings or engagement in work, school, or training at the end of the follow-up period. 
The program improved some survey-based measures of employment and job quality, but there 
were no statistically significant impacts on measures related to education or well-being. ! 


Impacts on Confirmatory Outcomes 


YAIP’s primary goal is to reconnect participants to the workforce, education, the military, or 
training programs, in hopes that engagement in these activities will ultimately improve partici- 
pants’ long-term economic prospects. The confirmatory outcomes analyzed in this report assess 
whether YAIP was achieving this goal at the end of the 30-month follow-up period. 


e Program and control group members were engaged in work, school, or 
training at similar levels at the end of the follow-up period, and the groups 
had similar earnings in the last year of follow-up. 


As shown in Table 2.1, program group members were slightly more likely than control 
group members to be employed or enrolled in an educational or training program at the end of 
the follow-up period, but the difference was not statistically significant.? Similarly, the program 


‘All estimates adjust standard errors for the following participant baseline characteristics: age; gender; race; 
whether the participant had a high school diploma or equivalency credential, had a disability, received food 
stamps, had limited literacy, had a child, was pregnant, or had impregnated someone at the time of study enroll- 
ment; whether or not the participant had ever been in foster care, been employed, enrolled in college, been ar- 
rested or convicted of a crime, run away from home, or been homeless before study enrollment; the number of 
months since previous job ended; and the cohort in which sample members were enrolled. 

This measure combines survey and administrative data. Point-in-time survey measures were combined with 
administrative data reflecting activity in the quarter in which participants were surveyed, which ranged from 10 
to 12 quarters after the quarter of random assignment. If a sample member had any activity in the quarter of 
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Table 2.1 


Confirmatory Impact Measures 


90 Percent 
Program Control Difference Confidence 
Outcome Group Group — (Impact) Interval 
Employment and education outcome (combined data sources) 
Currently employed, enrolled in school, or participating in 
vocational training* (%) 84.2 81.5 2.7 [-0.1, 5.5] 
Sample size (total = 2,031 1,272 759 
Administrative employment outcome 
Total earnings during last year of follow-up” ($) 8,131 7,637 494 [-84, 1,071] 


Sample size (total = 2,678 1,638 1,040 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires, 
postsecondary education data from the National Student Clearinghouse, and responses to the Subsidized and 
Transitional Employment Demonstration 30-month survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
aMeasure includes self-reported current employment and current enrollment in high school equivalency, 
college, or vocational training courses from the 30-month survey, and employment and postsecondary 
education in the quarter of survey administration from administrative data. Measure is restricted to the survey 
response sample. 
bMeasure is based on administrative data from the National Directory of New Hires. 


group earned slightly more than the control group in the last year of follow-up, but the difference 
was not statistically significant. The program group earned about $500 more than the control 
group, but the likely range of effects is from around -$80 to as much as about $1,070.* Although 
this predicted range of effects suggests that YAIP likely had a somewhat positive long-term im- 
pact on earnings, quarterly earnings results presented below show that the difference between 
program and control group earnings fades over time and disappears entirely in the final quarters 
of the follow-up period. 


This confirmatory analysis provides an overall assessment of YAIP’s performance rela- 
tive to two key program outcomes. However, an analysis of YAIP’s effects on other measures 


survey administration in the administrative data sources, or the sample member indicated current participation in 
education, employment, or training on the survey, they were considered currently engaged in this measure. 
3This measure is based on National Directory of New Hires data. 
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can provide useful insights for future program planning and policymaking. The following sections 
present YAIP’s impacts on measures of employment, education, and well-being after 30 months. 


Impacts on Employment and Earnings 


This section presents YAIP’s 30-month impacts on employment and earnings using administra- 
tive data from the National Directory of New Hires (NDNH), supplemented by subsidized earn- 
ings records from YAIP’s management information system and data from the 12-month and 30- 
month surveys of study participants. This combination of administrative and self-reported data 
sources provides a broad picture of YAIP’s effects on employment. NDNH data reflect jobs that 
were reported to the unemployment insurance system, as well as federal agency and state work- 
force agency employment; Y AIP subsidized earnings records reflect participation in YAIP sub- 
sidized internships; and survey data reflect all jobs that participants reported, which may have 
included jobs that were not captured by administrative data sources. 


e Program group members had higher earnings than control group mem- 
bers for several months after the end of the program’s internship period. 


Figure 2.1 shows quarterly employment and earnings during the evaluation period. 
Throughout the evaluation period, control group employment and earnings rose steadily. There 
were employment and earnings impacts in the early quarters of the follow-up period, during 
which program group members were most likely to be working in subsidized YAIP internships. 
However, employment effects faded in the fourth quarter after random assignment, and earnings 
effects faded in the seventh quarter after random assignment. The sustained earnings impact for 
three quarters after the employment impact faded suggests that, for several months after partici- 
pating in YAIP, program group members may have (1) earned higher wages, (2) worked more 
hours, or (3) been employed in jobs that were more stable, resulting in more consistent employ- 
ment. In other words, although YAIP did not lead to higher levels of employment, it may have 
led to higher-quality jobs for the period immediately following Y AIP. 


e Survey results suggest that YAIP improved some measures of employ- 
ment at the end of the follow-up period, and this improvement appears to 
have been driven by an increase in employment that was not covered by 
administrative data. 


Table 2.2 presents additional measures of employment and earnings from administrative 
(top panel) and survey (bottom panel) data sources. The top panel of the table highlights differ- 
ences between impacts during the first year after random assignment (corresponding to Quarters 
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Figure 2.1 


Employment and Earnings Over Time 
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SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New 
Hires and YAIP management information system subsidized earnings records. 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment 
characteristics. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 
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Table 2.2 


30-Month Impacts on Employment and Earnings 


Outcome 


Administrative employment outcomes 
Employment* (%) 

First year of follow-up 

Last year of follow-up 


Number of quarters employed 
First year of follow-up 
Last year of follow-up 


Employed in all quarters (%) 
First year of follow-up 
Last year of follow-up 


Total earnings ($) 
First year of follow-up 
Last year of follow-up 


Sample size (total = 2,678) 


Program 
Group 


95.2 
78.2 


2.6 
2.4 


28.1 
38.8 


6,674 
8,131 


1,638 


Self-reported employment outcomes (based on survey data) 


Ever employed (%) 
First year of follow-up 
Last year of follow-up 


Currently employed (%) 
At time of 12-month survey 
At time of 30-month survey 


Type of employment at 30-month survey” (%) 
Not currently employed 
Permanent 
Temporary, including day labor and odd jobs 
Other 


81.9 
87.3 


53.9 
65.0 


37.6 
52.3 
10.0 

0.1 
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Control 
Group 


66.4 
78.7 


1.6 
2.4 


13.7 
36.9 


3,247 
7,637 


1,040 


75.2 
85.3 


51.5 
60.5 


42.7 
47.1 
10.1 

0.2 


Difference 
(Impact) 


28.8 kK 
-0.4 


1.0 KKK 
0.0 


144 
1.9 


3,428 *** 
494 


6.7 KKK 
1.9 


2.5 
4.5 ** 


-5.1 * 
5.2 * 
0.0 
-0.1 


90 Percent 
Confidence 
Interval 


[26.6, 31.0] 
[-3.1, 2.2] 


[0.9, 1.0] 
[-0.1, 0.1] 


[11.9, 17.0] 
[-1.1, 4.9] 


[3,109, 3,746] 
[-84, 1,071] 


[3.7, 9.7] 
[-0.6, 4.5] 


[-1.2, 6.1] 
[0.9, 8.1] 


[-10.0, -0.2] 
[0.5, 10.0] 
[-0.3, 0.3] 
[-1.1, 0.9] 

(continued) 


Table 2.2 (continued) 


90 Percent 

Program Control Difference Confidence 

Outcome Group Group (Impact) Interval 
Hours worked per week at 30-month survey (%) 

More than 20 hours 52.5 49.0 3.5 [-0.3, 7.2] 

More than 34 hours 35.1 30.6 4.5 ** [0.9, 8.1] 

Hourly wage at 30-month survey (%) 

More than $8.00 55.7 49.9 5.8 ** [1.9, 9.7] 

More than $10.00 34.3 30.0 4.3 * [0.6, 7.9] 

More than $12.00 19.3 15.9 3.4 * [0.4, 6.4] 


Among those currently employed at 30-month survey* 


Hours worked per week 33.0 32.7 0.3 -- -- 

Hourly wage ($) 12.0 12.0 -0.1 -- -- 
Sample size 

12-month survey (total = 2,122) 1,325 797 

30-month survey (total = 2,031) 1,272 759 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires, YAIP 
management information system subsidized earnings records, and responses to the Subsidized and Transitional 
Employment Demonstration (STED) 12-month and 30-month surveys. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

aEmployment rates and earnings include both STED subsidized jobs and all other jobs covered by 
unemployment insurance. 

>To account for correlations between statistical tests of individual categories of a categorical outcome, 
significance for this measure was calculated using a Westfall-Young procedure. 

‘These measures are calculated among those employed at the time of the survey; they are therefore 
considered nonexperimental and are not tested for statistical significance. 


0 through 3 in Figure 2.1) and during the last year of the full 30-month follow-up period (corre- 
sponding to Quarters 6 through 9 in Figure 2.1). During the first year of follow-up, YAIP in- 
creased employment by nearly 30 percentage points and increased earnings by $3,428 (a 106 
percent increase in earnings compared with the control group).* Program group members were 
also employed for more quarters and more likely to be employed in all four quarters of the first 


‘Just as YAIP earnings were not reported to the NDNH, it is possible that control group members partici- 
pated in subsidized employment programs that did not report subsidies to the NDNH. Thus, it is possible that 
control group earnings were underestimated. 
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year of follow-up than their control group counterparts. In the last year of follow-up, there were 
no significant differences between groups on any administrative data-based measures. Although 
the administrative data found no earnings or employment impacts in the last year of follow-up (as 
shown in Table 2.2) or in the final quarter of the follow-up period (as shown in Figure 2.1), survey 
results suggest that program group members were more likely to be employed, working in per- 
manent jobs, working more hours per week, and earning higher wages at the end of the 30-month 
follow-up period.§ 


However, sample members in both research groups who were employed at the time of 
the survey worked an average of 33 hours per week and earned $12 per hour, suggesting that the 
differences in hours worked and wages among the full survey sample are primarily due to more 
program group members working rather than differences in hours worked or wages earned by 
working participants. Notably, there was not a statistically significant increase in program group 
members working over 20 hours per week, only in their working over 34 hours per week, sug- 
gesting that the difference stems from more program group members working in full-time rather 
than part-time jobs. 


The administrative and survey findings tell different stories about sample members’ em- 
ployment at the end of the follow-up period. While administrative findings suggest that the pro- 
gram had no long-term employment and earnings effects, survey findings suggest that the pro- 
gram had modest, positive effects on several measures of employment. An analysis of survey 
response bias shows little evidence of differences in response patterns between research groups, 
which suggests that YAIP increased employment that is detected by the survey but not captured 
by the NDNH.° Indeed, an analysis of occupations reported on the survey reveals that program 
group members are slightly more likely to report being employed in jobs that are not generally 
covered by the NDNH, such as domestic work, day labor, and babysitting. It is unclear what 
mechanism would increase uncovered employment for program group members, but many recent 
evaluations of employment programs have found a similar pattern, including evaluations of Fam- 
ily Rewards, Work Rewards, Enhanced Transitional Jobs programs, and YouthBuild.’ So, it 
seems that, in many cases, employment programs are increasing the proportion of participants 
who find employment in independent or informal work arrangements, a phenomenon that bears 
further investigation. 


5P-values for categorical variables such as type of employment were adjusted for multiple comparisons 
using a Westfall-Young procedure. 


SSee Appendix A for a review of the survey response bias analysis. 
Miller et al. (2016); Verma, Yang, Nufiez, and Long (2017); Redcross et al. (2016); Miller et al. (forthcom- 
ing). 
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Impacts on Other Outcomes 


There were no statistically significant impacts on outcomes in any other domains. Nonetheless, 
results from this analysis provide a glimpse of the experiences of young people in the sample. 
Most measures discussed in this section were calculated based on survey responses, with the ex- 
ception of measures of postsecondary education, which were calculated based on administrative 
data from the National Student Clearinghouse. 


Education 


In the education domain, as shown in Table 2.3, about 16 percent of each research group 
was enrolled in high school equivalency or vocational training courses at the end of the follow- 
up period, and just under one-quarter of sample members had enrolled in a postsecondary insti- 
tution since random assignment. Over one-fifth of the sample earned a high school diploma or 
equivalency certificate since random assignment. Notably, as shown in Table B.1, nearly half of 
sample members who did not have a high school credential at baseline earned one in the 30 
months after study enrollment, with the program group slightly more likely to earn a credential. 


e Control group members enrolled in postsecondary education at higher 
rates during the program period, but enrollment rates leveled out after 
the program ended. 


As noted in Chapter 1, postsecondary enrollment is one of YAIP’s key program out- 
comes, along with employment, military enlistment, and advanced training. Figure 2.2 shows 
postsecondary enrollment over the course of the evaluation, calculated based on administrative 
records. Control group members enrolled in postsecondary courses at statistically significantly 
higher rates than program group members during Quarters 0 and 1 (that is, the program period), 
as well as Quarter 2, the quarter directly after most program group members had stopped partici- 
pating in YAIP. In Quarters 3 through 9, postsecondary enrollment levels were similar in the 
program and control groups, with slightly more program group members enrolled in all but one 
quarter. In some ways, this trend is a less extreme inverse of the employment trend presented in 
Figure 2.1: It seems that, in the absence of Y AIP, some control group members opted to enroll in 
college. However, the early statistically significant decrease in program group postsecondary en- 
rollment did not lead to differences in enrollment or degree receipt during the full follow-up pe- 
riod. 


Notably, both groups’ postsecondary enrollment rates trend upward throughout the fol- 
low-up period, just as control group employment rates trend upward throughout the first eight 
quarters of the follow-up period. This finding is further evidence of two important points dis- 
cussed in the interim report: (1) disconnectedness is often a temporary and inconstant state, and 
(2) by taking the step to seek out YAIP services, members of both research groups were likely 
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Table 2.3 


30-Month Impacts on Education and Training 


Program Control Difference 


Outcome (%) Group Group (Impact) 

Self-reported education outcomes (based on survey data) 

Currently participating in education and training 16.5 15.5 1.1 
GED or high school diploma classes* 6.0 6.5 -0.4 
Vocational training 11.6 10.1 1.5 


Earned a high school diploma or equivalency certificate 


since random assignment 22.0 21.7 0.3 
Has a professional license or certification 31.7 31.0 0.7 
Sample size (total = 2,031 1,272 759 


Administrative education outcomes 


Enrolled in a postsecondary institution since random assignment 22.7 24.8 -2.1 
Enrolled in four-year college 8.3 8.4 -0.1 
Enrolled in two-year college 16.2 17.6 -1.4 

Earned a postsecondary degree since random assignment? 2.7 2.0 0.6 

Sample size (total = 2,678) 1,638 1,040 


90 Percent 
Confidence 
Interval 


[-1.6, 3.8] 
[-2.2, 1.3] 
[-0.9, 3.9] 


[-2.5, 3.1] 


[-2.7, 4.2] 


[-4.7, 0.4] 
[-1.8, 1.7] 
[-3.7, 1.0] 


[-0.4, 1.6] 


SOURCES: MDRC calculations based on responses to the Subsidized and Transitional Employment Demonstration 


30-month survey and postsecondary education data from the National Student Clearinghouse. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 


Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


aGED = General Educational Development. 
>Measure includes associate's, bachelor's, and master's degrees. 


motivated at the time of study enrollment to make a change in their lives. For these reasons, one 
would expect to see growth in levels of engagement in work and school for both groups over 


time. 
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Figure 2.2 


Postsecondary Enrollment Over Time 
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SOURCES: MDRC calculations based on postsecondary education data from the National Student 
Clearinghouse. 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


Well-Being 


e Sample members mirror the general population on measures of personal 
well-being, despite being worse off on most measures of economic well- 
being. 


In the economic well-being domain, as shown in Table 2.4, about one-third of sample 
members experienced a financial shortfall in the final year of follow-up, and one in six sample 
members experienced food insufficiency in the month before the 30-month follow-up survey. 
The sample fared poorly on various measures of economic well-being compared with the general 
population. For example, the sample’s rate of health insurance coverage is much lower than that 
of the general population: 67 percent of the sample had health insurance at the end of the follow- 
up period, whereas 87 percent of adults ages 19 to 25 had health insurance coverage in 2016, 
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Table 2.4 


30-Month Impacts on Economic and Personal Well-Being 


Outcome 


Economic well-being outcomes (% 
Experienced financial shortfall in past 12 months 
Could not pay rent or mortgage 
Evicted from home or apartment 
Utility or phone service disconnected 
Could not afford prescription medicine 


Experienced food insufficiency in prior month 
Homeless or lived in a shelter in prior month 


Had health insurance coverage in prior month 
Health insurance coverage was employer based 


Personal well-being outcomes 
Currently in good, very good, or excellent health (%) 


Experienced serious psychological distress in prior month* (%) 
Score on self-esteem scale” 
Score on social support scale® 


Overall happiness® (%) 


Very happy 
Pretty happy 


Not too happy 


Sample size (total = 2,031) 
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Program 
Group 


32.0 
20.8 
3.1 
15.8 
9.0 


17.8 
3.8 


67.0 
12.5 


86.9 
8.4 
3.3 


4.0 


26.8 
59.4 
13.9 


1,272 


Control Difference 


Group 


31.8 
18.0 

2.3 
16.3 
10.9 


16.3 
4.4 


67.1 
12.9 


88.1 
8.5 
3.3 


3.9 


26.3 
61.0 
12.7 


759 


(Impact) 


0.2 
2.8 


0.0 


0.4 
-1.7 
1.2 


90 Percent 
Confidence 
Interval 


[-3.3, 3.6] 
[-0.1, 5.8] 
[-0.5, 2.0] 
[-3.2, 2.4] 
[-4.2, 0.3] 


[-1.4, 4.3] 
[-2.1, 0.8] 


[-3.6, 3.4] 
[-2.9, 2.2] 


[-3.8, 1.3] 
[-2.2, 2.1] 
[0.0, 0.0] 


[-0.1, 0.1] 


[-2.7, 3.6] 
[-7.3, 3.9] 
[-2.9, 5.3] 


(continued) 


Table 2.4 (continued) 


SOURCE: MDRC calculations based on responses to the Subsidized and Transitional Employment Demonstration 
30-month survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

aA score of 13 or higher on the Kessler-6 Scale (K-6) is used here to define serious psychological distress. 
Response categories range from 1 to 4, where higher scores indicate higher levels of psychological distress. As a 
result of minor differences between the scale used to administer the K-6 in the survey and the standard K-6, the 
percentages presented in this table may slightly underestimate the incidence of serious psychological distress 
among the YAIP sample. 

>The Rosenberg Self-Esteem Scale is a 10-item scale that assesses feelings of self-esteem. Response 
categories range from 1 to 4, where higher scores indicate higher levels of self-esteem. The 10 items are 
averaged. 

‘The Medical Outcomes Study Social Support Survey is a seven-item scale that assesses the types of social 
support available to respondents. Response categories range from 1 to 5, where higher scores indicate higher 
levels of social support. The seven items are averaged. 

‘To account for correlations between statistical tests of individual categories of a categorical outcome, 
significance for this measure was calculated using a Westfall-Young procedure. 


when most follow-up surveys were fielded. Sample members with health insurance were cov- 
ered by a variety of private and public providers and were often covered by more than one form 
of health insurance. Notably, three-quarters of sample members with health insurance coverage 
are covered, at least in part, by public insurance, compared with 23 percent of adults ages 19 to 
25,” 


Measures of personal well-being, shown in Table 2.4, reveal that the sample is about as 
healthy and happy as the overall population. Nearly 90 percent of the sample reported being in 
good, very good, or excellent health and being very or pretty happy at the time of the 30-month 
follow-up survey. Assessments of health and happiness in this sample are comparable to a na- 
tional sample of 18- to 34-year-olds who responded to the General Social Survey in 2014. '° Under 
9 percent of the YAIP sample experienced serious psychological distress in the month before the 
survey. This result is consistent with other studies using the Kessler-6 measure of psychological 
distress.!! Thus, despite experiencing more economic hardship, the sample is similar to the gen- 
eral population on measures of mental and physical health. 


®Bamett and Berchick (2017). 

°Barnett and Berchick (2017). 

!0Smith, Marsden, Hout, and Kim (2014). 

‘1Prochaska et al. (2012); Shafer, Koenig, and Becker (2017). 
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Conclusion 


The results presented above tell the story of a group of young people who are motivated to make 
a change in their lives and a program that facilitates substantial short-term employment improve- 
ments that diminish over time. There were modest differences in survey-based employment 
measures at the time of the 30-month follow-up survey, including current employment and the 
type of job worked. These improvements appear to be driven by an increase in employment in 
jobs not covered by the NDNH. Although uncovered jobs can range from low-wage, inconsistent 
jobs such as day labor to higher-wage, relatively consistent jobs such as independent consulting, 
the types of uncovered jobs reported by survey respondents tend to be on the low-wage end of 
the spectrum, specifically jobs such as babysitting and domestic work. Quarterly employment and 
postsecondary enrollment rates suggest that some participants may have opted out of postsecond- 
ary classes because they were participating in YAIP, a trade-off effect typical of youth employ- 
ment programs. However, enrollment rates evened out after the program ended, and there were 
no lasting, negative postsecondary enrollment effects during the study period. Finally, sample 
members mirror the larger population on measures of personal well-being, despite being worse 
off on many measures of economic well-being. 
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Chapter 3 


Impact Variation 


Chapter 2 describes the overall effects of YAIP, across all participants and locations in the study. 
However, YAIP may work better for some types of participants or in some sites than others, and 
these differences would not be reflected in the overall effects. This chapter summarizes an anal- 
ysis of how YAIP’s effects may vary across participant subgroups — specifically, gender, edu- 
cation level, length of disconnection, and cohort — program locations, and program characteris- 
tics. A more detailed description of the analysis can be found in Appendix B. 


Variation by Participant Characteristics 


Findings from the interim report showed some differences in impacts across types of young peo- 
ple, with larger effects on employment for male and less-educated sample members, members of 
Cohorts 1 and 2, and sample members who had been disconnected longer. 


e YAIP’s longer-term effects do not vary much across different types of 
young people. 


As shown in Appendix Tables B.1 to B.4, at 30 months, there was not much variation in 
effects across types of young people. Most differences observed were in education-related out- 
comes, which was not where the program had its effects. However, effects on earnings in the last 
year of follow-up were larger for women. 


Variation by Location 


Just as impacts may vary according to sample members’ sociodemographic characteristics, it is 
possible that some YAIP locations performed better than others relative to their counterfactual 
alternatives. Therefore, it is useful to (1) estimate the amount of impact variation across locations 
and (2) examine the cross-location distribution of program effects to understand the range of im- 
pacts YAIP can expect to produce.' This analysis tested variation in location-level impacts for 


‘Twelve providers administered YAIP in 13 locations. This analysis assesses variation across individual 
locations rather than individual providers, as contextual factors, treatment contrast, and participant characteristics 
are expected to vary across locations. 
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both confirmatory measures, as well as for earnings in the first year of follow-up, in order to 
examine variation in earnings across locations during the program period.” 


e Despite statistically significant cross-location variation in impacts on 
Year 1 earnings, there was little variation in impacts on confirmatory out- 
comes at the end of the follow-up period. 


Program effects on earnings varied widely across locations during the program period 
but did not vary during the last year of follow-up. As shown in Appendix Figure B.1, during the 
program period, impact estimates ranged from $2,194 to $4,684 in magnitude — a $2,490 range 
of impacts across locations — and there was a large, statistically significant effect on earnings at 
each location during the program period. These results corroborate the implementation study find- 
ing that each site implemented the model with fidelity and confirm that, although treatment con- 
trast varied significantly across locations, each site significantly increased earnings compared 
with local alternatives. At the end of the follow-up period, each location had the same estimated 
effect on earnings of about $500. Similarly, current employment or enrollment in education and 
training did not vary across locations: Each location had a 2.7 percentage point estimated effect 
on this measure.? 


Variation by a Key Program Characteristic 


YAIP’s effectiveness may be associated with factors related to the way the program was imple- 
mented in each location; thus, measures of location characteristics may help explain variability in 
effects, even when a statistical test of variation shows no differences in cross-location impacts. 
Therefore, this analysis tested whether one characteristic — the presence of a job developer on 
staff — was associated with program effects.* The presence of a full-time unsubsidized job de- 
veloper on staff at a YAIP location was not associated with program impacts. 


“Year 1 earnings impacts can serve as a measure of treatment contrast, capturing both the amount of subsi- 
dized employment a participant worked and the relative quality of a participant’s job following YAIP, which 
may have been a result of strong job placement services. See Appendix B for a description of analytical methods 
and further discussion of results. 


There was not enough variation in effects on earnings in the last year of follow-up or engagement in em- 
ployment, education, or training at the end of the follow-up period to calculate location-level effects. In other 
words, the best estimate of program effects at each location was equal to the estimate of effects across all loca- 
tions. 


‘This analysis only examined one program characteristic because of statistical power considerations. Pro- 
vider staff considered this measure to be one of the most important location-level factors that varied across loca- 
tions. 


28 


Conclusion 


YAIP’s long-term effects were fairly consistent across types of participants and types of provider 
locations, and effects did not vary by whether programs employed a full-time job developer. 
However, impacts on earnings during the first year of follow-up varied significantly across loca- 
tions, suggesting that even under the best circumstances (that is, when the services a location 
provides contrast most starkly with local alternatives), YAIP may not lead to long-term improve- 
ments in key outcomes. 
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Chapter 4 


Cost Analysis 


As described in Chapter 1, the 12 YAIP providers that participated in the study offered program 
group members a 10- to 12-week paid internship, along with various other services, including 
job-readiness workshops and activities; individual support, counseling, and assessments; case 
management; and follow-up services. 


This chapter presents the costs of YAIP services and the wages provided to program 
group members who participated in YAIP. It begins with a review of the data sources, followed 
by a description of the methodology used to estimate the costs. It then presents an estimate of the 
costs of YAIP per program group member and compares these costs to those of other programs 
offering comparable services. 


Data Sources 


The following data sources, described in Chapter 1, were used to calculate the cost estimates: 
Mayor’s Office for Economic Opportunity (NYC Opportunity) and Young Men’s Initiative 
(YMI) cost reports, provider budget reports and salary information, New York City Department 
of Youth and Community Development cost estimates, staff time study data, and program partic- 
ipation data. 


Methodology 


The research team selected the 2014 fiscal year for examining the costs of the program because 
it was a period when all three cohorts received services. All three study cohorts completed Phases 
1 and 2 (orientation and internship) services during this year. Two of the three study cohorts also 
received Phase 3 (nine-month follow-up) services during this period; Cohort 3 began Phase 3 at 
the beginning of the 2015 fiscal year. All costs were inflated to 2016 dollars. 


Operating Costs 


Operating costs are the day-to-day costs of operating the program and include expendi- 
tures on staff salaries, fringe benefits, overhead, and administration. 


The research team collected YAIP staff salary information and used results from the staff 
time study — in which YAIP staff recorded how they spent their time during two separate weeks 


The New York City budget cycle runs from July 1 to June 30. 
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in 2014 — to allocate their salaries to the YAIP program. While most staff members were as- 
signed full time to the YAIP program, some staff worked on other programs. For those who 
worked part time on YAIP, a percentage of their salary was allocated to Y AIP based on the per- 
centage of their time they spent on YAIP activities. The YAIP salaries were further allocated to 
specific YAIP activities based on the staff time study results. The activities included application 
and outreach, case management, workshops, internship placements, worksite management, post- 
internship placements, job search assistance, and administrative activities (for example, payroll 
review, program administration, and staff meetings). For case management and job search, staff 
members further distinguished their time between working with clients currently active in Phases 
1 and 2 and those receiving follow-up services (Phase 3). 


The staff salaries were marked up to include fringe benefits, overhead, and other admin- 
istrative costs as estimated from the provider budget reports. 


The research team used the YAIP costs during the 2014 fiscal year and results from the 
staff time study to calculate the average monthly cost of providing YAIP services to clients active 
in Phases 1 and 2 and the average monthly cost of providing Phase 3 services. Each of these 
monthly unit costs was multiplied by the average number of months that program group members 
spent in the respective phases, as recorded in the program’s management information system, to 
obtain the average cost per program group member in Phases 1 and 2 and the average cost per 
program group member in Phase 3. 


Support Services 


Support services provided to YAIP participants were listed separately in the provider 
budgets. These costs included transportation and other support services. A similar methodology 
that was used to estimate operating costs was used to estimate support services, first estimating 
the monthly cost of support services per program group member and multiplying this cost by the 
average number of months spent in the program. 


Wages and Payroll Costs 


Data on participants’ subsidized earnings came from the Y AIP management information 
system. Information on payroll costs — payroll processing fees, workers’ compensation, and pay- 
roll taxes — came from NYC Opportunity and YMI reports. The subsidized earnings were 
marked up to include the payroll costs paid by the program. 


YAIP Program Costs 


The costs of providing Y AIP program services are divided into three types: (1) the operating costs 
of YAIP, which includes all staff salaries, fringe benefits, overhead, and administrative costs; (2) 
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Table 4.1 


Estimated YAIP Costs per Program Group Member 
(in 2016 dollars) 


Services Total 


Operating costs 
Phases 1 and 2 


Cost per month of service ($) 805 

Average months of service 3.0 

Program service cost per program group member ($) 2,447 
Phase 3 

Cost per month of service ($) 306 

Average months of service 3.0 

Program service cost per program group member ($) 927 
Operating cost per program group member ($) 3,374 


Support service costs 


Cost per month of service ($) 19 
Average months of service 6.1 
Support service cost per program group member ($) 115 


Wages and payroll costs ($) 


Subsidized wages per program group member 1,741 
Payroll costs per program group member 201 
Wages and payroll costs per program group member 1,942 
Total cost per program group member ($) 5,431 


SOURCE: MDRC calculations based on YAIP program cost data. 


the support services provided to program group members; and (3) the subsidized wages and pay- 
roll costs. As shown in Table 4.1, the cost of providing the YAIP program services averaged 
$3,374; $2,447 per program group member in Phases 1 and 2 (orientation and internships) and 
$927 per program group member for Phase 3 follow-up services. The program paid for transpor- 
tation and other work-related costs to help participants attend the program, which averaged $115 
per person. The subsidized wages and payroll costs averaged $1,942 per person. The total cost 
per program group member was $5,431. 


33 


Figure 4.1 


YAIP Service Costs by Component 


Administration 

Case management 
Workshops 
Application/outreach 
Worksite management 
Post-internship placements 


Internship placements 


Job search assistance 


SOURCE: MDRC calculations based on YAIP program cost data. 


About 62 percent of the total costs were spent on program operations. Figure 4.1 shows 
the distribution of these costs, based on how staff members spent their time from the staff time 
study. As this figure shows, 41 percent of these costs were spent on administrative functions, 16 
percent on case management, 13 percent on application and outreach activities, and 14 percent 
on educational workshops. The remaining 18 percent was related to job search, job development, 
worksite management, and post-internship placements.* While most staff members were not ad- 
ministrative, they did record time on administrative functions such as attending staff meetings, 
collecting documentation and outcome information, reviewing time sheets, and other program 
administration activities. 


YAIP Compared with Other Programs 


New York City has a number of employment programs for young people, so it is not surprising 
that over half (53 percent) of control group members reported receiving help finding or keeping 


“Distributions may not sum to 100 percent due to rounding. 


a job. Included in this measure, 11 percent of control group members reported receiving help 
finding unpaid work experience or internships. Almost half (46 percent) of control group mem- 
bers reported receiving advice or support from a staff member at an agency. 


Though the participant survey captured information on the extent to which control group 
members received services similar to program group members, it was not known where control 
group members received their services. Thus, the analysis could not estimate the cost of services 
provided to control group members and the net cost of YAIP — the amount over and above what 
was spent on the control group. 


It is useful to compare the cost of YAIP with other programs that have been evaluated. 
Table 4.2 provides estimates of other programs, some of which served young people, though with 
a different range of services, and others that may have provided similar services (transitional jobs 
or internships), though to a different population. 


The programs that provided similar services — transitional employment and job place- 
ment assistance — but to a different population include the Transitional Jobs Reentry Demon- 
stration, which targeted men who had recently been released from prison in Chicago, Detroit, 
Milwaukee, and St. Paul, and the Transitional Work Corporation, which served recipients of 
Temporary Assistance for Needy Families in Philadelphia. The costs for these programs, $4,805 
and $4,201 (in 2016 dollars), respectively, were similar to or a little lower than the cost per person 
of YAIP.° Similar to YAIP, these programs spent more on program operations than wages and 
support services. The costs of the Enhanced Transitional Jobs Demonstration, which also served 
an older population but provided more services than YAIP, averaged $8,204 across the seven 
sites. Programs that served a similar population — disconnected youth — but provided more 
education and training or enhanced services, included Jobstart, Project Rise, and Youth Transition 
Demonstration. The costs of these programs were significantly higher, ranging from $6,854 to 
close to $10,000.5 


3Redcross et al. (2010); Bloom et al. (2009). 
4Barden et al. (Forthcoming). 
°Cave, Bos, Doolittle, and Toussaint (1993); Manno, Yang, and Bangser (2015); Fraker et al. (2014). 
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Table 4.2 


Comparison of YAIP Costs to Other Programs (in 2016 dollars) 


Program 


YAIP 


Enhanced 
Transitional Jobs 
Demonstration 


Transitional Jobs 
Reentry 
Demonstration 


Transitional Work 
Corporation 


Jobstart 


Project Rise 


Youth Transition 
Demonstration 
(average across 
six sites) 


Services 


Job-readiness workshops, 


internships, and job 
placement assistance 


Subsidized jobs, case 


management, work-related 


supports, job placement 


assistance, and enhanced 
services depending on site 


(for example, enhanced 
support, child support 
incentives) 


Transitional jobs, support 
services, and job placement 


assistance 


Subsidized jobs, case 


management, work-related 


supports, job placement 
assistance 


Education and training, 
support services, job 
placement assistance 


Classroom education, part- 
time internships, assistance 


with transitioning into 


unsubsidized employment 
or postsecondary education 


Paid employment and 


enhanced work incentives, 
youth empowerment, family 


supports, benefits 
counseling 


Sample 


Low-income young 
people between ages 
16 and 24 who were 
neither in school nor 
working 


Individuals recently 
released from prison or 
noncustodial parents 
who owed child support 


Men over 18 years of 
age who had been 
released from state 
prison 


TANF recipients in 
Philadelphia 


Low-income young 
people between ages 
17 and 21 who lacked 
basic skills and who 
dropped out of school 


Young people between 
ages 18 and 24 who 
lacked a high school 
degree 


Young people with 
disabilities between 
ages 14 and 25 


Program 
Operations ($) 


3,374 


5,449 


2,639 


3,199 


8,459 


5,131 


7,818 


Wages and 
Support 
Services ($) 


2,057 


2,755 


2,166 


1,002 


1,500 


1,723 


344 


Total ($) 


5,431 


8,204 


4,805 


4,201 


9,959 


6,854 


8,162 


SOURCES: MDRC calculations from YAIP, Enhanced Transitional Jobs Demonstration, Transitional Jobs Reentry 
Demonstration, Transitional Work Corporation, Jobstart, Project Rise, and Youth Transition Demonstration program 


cost data. 
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Chapter 5 


Conclusion 


This report presents final impact and cost analysis findings from a random assignment study of 
New York City’s Young Adult Internship Program (YAIP), one of several programs being eval- 
uated as part of the Subsidized and Transitional Employment Demonstration (STED). This chap- 
ter summarizes the final (30-month) impacts of the program and offers some considerations for 
the design of programs similar to Y AIP based on these findings. 


Summary of Findings 


The research team assessed YAIP’s impacts using data from three key sources: employment and 
earnings data from the National Directory of New Hires, 30-month survey data, and postsecond- 
ary enrollment data from the National Student Clearinghouse. This report tracked study partici- 
pants’ outcomes for 30 months after random assignment to assess differences in outcomes be- 
tween program and control group members. 


e YAIP led to large improvements in employment and earnings during the 
program period. At the end of the follow-up period, however, participants 
did not engage in YAIP’s key outcomes — education, employment, the 
military, or training — at a higher rate than they would have in the ab- 
sence of the program, and they did not earn more money. 


As shown in the interim report, YAIP increased annual earnings by about $3,400 in the 
first year of follow-up, more than doubling the annual income of program group members com- 
pared with their control group counterparts. For low-income young people living in a city with a 
high cost of living, this increase in income is substantial, and it is indicative of YAIP’s success in 
achieving its primary short-term goal of getting out-of-work young people rapidly employed in 
temporary jobs. However, employment and earnings effects diminished over time, and even 
groups that benefited the most from YAIP’s services in the short term did not retain many long- 
term benefits. During the program period, populations with larger barriers to employment at study 
enrollment (that is, those without a high school credential or those who were disconnected for a 
longer period) benefited the most from the program, but those benefits did not last through 30 
months of follow-up. Further, program group members in the YAIP locations that provided the 
strongest contrast of services relative to their local alternatives, as measured by earnings, did not 
perform better on key outcomes than their control group counterparts 30 months after the program 
began. However, YAIP increased survey-based employment at the end of the follow-up period, 
indicating that program group members may have been more likely to hold jobs in the informal 
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economy or as independent contractors. YAIP did not lead to any statistically significant effects 
in the education or well-being domains. 


e YAIP struggled to connect young people to unsubsidized employment or 
postsecondary education after the program ended, and both research 
groups were employed or enrolled in school at similar rates throughout 
the remainder of the follow-up period. 


After the program ended, the program group’s overall employment level dropped to that 
of the control group. The program group’s quarterly employment rate grew slowly in lockstep 
with control group employment rates for the next five quarters before both groups leveled out at 
about 61 percent employment for the final three quarters of the follow-up period. Importantly, 
this trend means that the control group doubled their quarterly employment rate in two years, 
supporting the hypothesis that people seek out programs such as YAIP when they are motivated 
to work. Quarterly postsecondary enrollment rates were similar between the two research groups 
after the program ended. 


Though the evaluation did not reveal much in the way of long-term impacts, the results 
tell a fascinating story about the resilience of a relatively job-ready subset of disconnected youth 
and the job market prospects for young adults with barriers to employment. Recall that 77 percent 
of the program group participated in a YAIP subsidized internship, and an impressive 95 percent 
of the program group worked during the first year of follow-up, suggesting that most of those 
who tured down the opportunity to participate in a YAIP internship were able to find jobs on 
their own. Two-thirds of the control group worked during the same period, either finding jobs on 
their own or, less often, through another subsidized employment or workforce development pro- 
gram. These figures depict a group of young people who are highly motivated to work but can 
have trouble finding and holding jobs on their own. 


e YAIP is at the lower end of the spectrum of costs for programs serving 
disconnected youth or providing transitional employment services. 


This report also includes a cost analysis, which used program budget, staff time, and par- 
ticipation data to estimate the cost of YAIP per program group member. The cost analysis found 
that YAIP costs a total of $5,431 per program group member, including subsidized wages, which 
is at the lower end of a group of similar programs ranging in cost per program group member 
from about $4,200 to $10,000. 


Program and Policy Implications 


The findings from this study show that, despite YAIP’s success in connecting young people to 
internships, the YAIP model may need to change in order to improve participants’ long-term 


38 


economic prospects. The findings presented in this report offer some insight on where Y AIP and 
programs similar to it may choose to focus their resources. 


e This evaluation builds on evidence from previous evaluations that sug- 
gests that providing light-touch employment and personal development 
services alongside temporary subsidized jobs that are unlikely to turn into 
permanent positions is not enough to improve participants’ long-term 
employment prospects. 


YAIP is not unique in its struggle to help participants make the transition into unsubsi- 
dized employment or postsecondary education; few rigorously evaluated subsidized employment 
programs have met this challenge. Many subsidized employment programs are designed based 
on a similar theory of change: Through job-readiness training and temporary work experience, 
participants will get the jumpstart they need to reconnect to the world of work, or in the case of 
youth programs to the worlds of work and school. However, this theory of change is based on a 
set of assumptions that are not supported by results from this evaluation — specifically, that there 
are jobs in the economy for young people with limited work experience and educational attain- 
ment, and that short-term employment programs can help participants gain the experience to qual- 
ify for, motivate them to apply for, and provide them with the work skills to maintain these jobs. 
This evaluation shows that, even with high levels of participation throughout the program, the 
program group gained few longer-term employment advantages over the control group, which 
belies the theory of change and raises the question: What works to improve employment prospects 
for more job-ready disconnected youth? 


Perhaps the diversity of the disconnected youth population calls for a variety of ap- 
proaches to reconnecting young people to school and work. Perhaps a three-month intervention 
is not long enough to promote long-term employment gains. Perhaps there is a mismatch between 
the skills and experience young people gain from a program such as YAIP and the skills and 
experience in demand in the workforce. If so, new approaches to workforce development pro- 
grams — such as apprenticeships and demand-driven training — may offer a solution. However, 
these approaches have not been tested with disconnected youth populations, who may need a 
program such as YAIP to prepare them for and connect them to these more rigorous programs. 
Perhaps participants lack the motivation, time, or support needed to apply for unsubsidized work 
or education programs while they are working in a subsidized job. Or, perhaps YAIP does not 
provide adequate tools for participants to engage in job searches on their own after the program 
ends, in which case programs should consider a different approach to unsubsidized job develop- 
ment and enrollment in education and training. 
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e A stronger focus on the transition from subsidized work to employment 
or school might help sustain the program’s short-term advantages in the 
long term. 


As evidenced by the steep drop in employment after the program ends and the lack of 
many longer-term impacts on employment or education, YAIP struggles to connect its partici- 
pants with work or school after the program ends beyond what these young people would have 
engaged in on their own. The transition from YAIP to potential unsubsidized employment or 
education is a crucial one for participants, as a failure to capitalize on the momentum that partic- 
ipants gain over the course of the program leads to a new period of disengagement for some 
participants, which could reduce the benefits, both economic and psychological, associated with 
the program. 


In order for YAIP to help participants in the longer term, the program might consider 
strengthening its focus on facilitating successful transitions from subsidized employment to un- 
subsidized employment or quality education and training programs. As a first step, given the fact 
that only 16 percent of those who worked in a subsidized internship were hired for permanent 
positions by their worksites, YAIP might consider making the possibility of post-internship hire 
a firmer criterion when developing worksites. Further, the lack of an association between the 
presence of a job developer on staff and employment impacts indicates that job developers may 
need more training and resources to do their jobs effectively. 


On the education and training front, YAIP might consider further developing relation- 
ships with postsecondary institutions and community-based organizations that provide high 
school equivalency credential preparation and certificate training to facilitate a seamless transition 
between a YAIP internship and education and training programs. However, getting students en- 
rolled is only half the battle: Disconnected youth face steep barriers to persistence in these pro- 
grams and may need support to remain enrolled. Connecting Y AIP participants with training pro- 
grams that have strong support services and evidence-based student retention programs — such 
as the City University of New York’s Accelerated Study in Associate Programs — may increase 
their odds of persisting, graduating, and ultimately gaining the job market benefits associated with 
a college degree or industry-recognized credentials. ! 


Where Is YAIP Now? 


In recent years, the Department of Youth and Community Development (DYCD) has made 
changes to YAIP in line with many of the recommendations discussed above. First, DYCD has 


‘Scrivener et al. (2015). 
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encouraged providers to broaden their approach to education and training, including incorporat- 
ing credential training into the subsidized portion of the program and adjusting participants’ 
schedules to accommodate high school equivalency programming. Second, DYCD has integrated 
more career exploration activities, such as career panels, employer visits, and mentoring, in order 
to help shift the focus from getting jobs to building careers. Finally, recognizing the need to pro- 
vide additional support to participants with steeper barriers to employment, DYCD launched a 
pilot program for young adults with experience in the foster care or juvenile justice systems that 
includes intensive case management. 


Next Steps 


This report is the final one in the Y AIP evaluation, but a future report will further analyze findings 
across 13 transitional jobs programs evaluated by MDRC as part of the STED evaluation and the 
Enhanced Transitional Employment Demonstration, funded by the U.S. Department of Labor. 
This future report will synthesize findings across these 13 programs to determine what cross- 
cutting lessons can be learned to inform the development of future employment programs. 
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Appendix A 


Survey Response Bias Analysis 
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This analysis examines the survey response for the last of three surveys administered as part of 
the YAIP evaluation, which was administered at roughly 30 months post-random assignment.! A 
subset of the full research sample completed each survey; therefore, it is possible that those who 
participated in the surveys are not representative of the full research sample, which could intro- 
duce bias into the estimates produced from the survey data. It is likely that the survey response 
sample differs slightly from the full research sample in terms of socio-demographic characteris- 
tics, as certain characteristics such as age, gender, and stability are generally associated with sur- 
vey response. The main concern is differences between program and control group respondents: 
If there are differences between the type of program group member who responds to the surveys 
and the type of control group member who responds to the surveys, impact estimates based on 
the surveys may be biased. 


Overall, the administration of the survey was fair, with a response rate of about 76 per- 
cent, and most interviews were completed on time (that is, within the survey fielding window of 
four months). There are a few small differences in the socio-demographic characteristics of the 
survey respondents compared with nonrespondents, which, for the reasons described above, is a 
typical finding of survey response analysis. However, the baseline characteristics of the members 
of the two research groups are similar within the survey response sample. In addition, program 
impacts on administrative outcomes among survey respondents are comparable to those estimated 
for the full research sample, though some differences did arise, indicating that there was limited 
survey response bias. 


Response Differences 


To test whether survey respondents differed from nonrespondents, the research team compared 
the socio-demographic characteristics of these two groups. As shown in Appendix Table A.1, 30- 
month survey respondents differed significantly from nonrespondents on a few characteristics. 
Specifically, respondents were more likely to be female, to be black, or to have had a limited 
work history and were less likely to be Hispanic or “Other, non-Hispanic,” to have had a serious 
barrier to employment, or to have ever been arrested. Respondents were also slightly younger. 
These response patterns are similar to response patterns for the 12-month survey. 


‘The two other surveys used in this evaluation, administered at roughly 4 and 12 months after random as- 
signment, showed few signs of response bias. 
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Appendix Table A.1 


Selected Baseline Characteristics of Survey Respondents 
and Nonrespondents, 30-Month Survey 


Characteristic Respondents Nonrespondents 
Average age (years) 20.6 20.8 * 
Female (%) 53.0 44.4 *** 
Race/ethnicity (%) ‘ 
Hispanic 35.3 39.1 
Black, non-Hispanic 59.5 53.7 
Other, non-Hispanic 5.2 real 
Ever employed (%) 72.1 70.5 
Worked in last three months (%) 28.9 33.8 ** 
Has children (%) 20.0 19.8 
Has high school diploma or GED certificate (%) 62.5 59.1 
Serious barrier to employment* (%) 41.1 45.9 ** 
Receives public assistance (%) 26.4 25.9 
Ever arrested (%) 24.1 32.4 *** 
Sample size 2,031 647 


SOURCES: MDRC calculations based on data from MDRC's random assignment system and the YAIP 
management information system. 


NOTES: Statistical significance levels are indicated as follows: *** =1 percent; ** = 5 percent; * = 10 percent. 
@This measure includes ever having run away from home, been homeless, or lived in unstable housing; 
history of foster care; limited literacy or math skills; mental or physical disability; previous criminal conviction; 

and pregnancy or having a child. 


Because comparison of a series of characteristics is susceptible to false positives, the re- 
search team conducted a global test of the relationship of these characteristics to response status. 
This test is conducted by estimating a regression model predicting survey response, and the test 
statistic reported for each characteristic indicates whether that characteristic has a statistically sig- 
nificant association with survey response, controlling for the other characteristics. The joint test 
indicates whether the characteristics collectively have a statistically significant association with 


46 


survey response. A few characteristics — specifically, gender, race, limited work history, ever 
arrested, and age — have significant effects. The overall joint test is also statistically significant, 
indicating that response status for this survey can be predicted by these characteristics. These 
associations may indicate some level of response bias, but this bias would primarily affect out- 
come estimates rather than impact estimates, as the bias affects both program and control group 
members. 


Of primary concern in an impact analysis are differences between research groups within 
the respondent sample. If respondents’ socio-demographic characteristics vary by research group, 
the impact estimates may not reflect true differences between groups. Thus, the research team 
compared socio-demographic characteristics of survey respondents by research group. As shown 
in Appendix Table A.2, survey respondents were similar across research groups, and the joint test 
of the association between socio-demographic characteristics and research groups for survey re- 
spondents was not significant. Thus, the impact estimates presented in the report are unbiased in 
terms of socio-demographic characteristics. 


Impact Differences 


Another way to assess possible bias from survey response is to examine differences in impacts 
measured with administrative data between the survey respondent sample and the full research 
sample. If the differences between the program and control groups in the survey respondent sam- 
ple are not similar to those observed for the full research sample, it would indicate that the survey 
response sample is not representative; thus, survey-based impacts may be biased. Appendix Table 
A.3 presents selected 30-month impacts based on administrative data for the research and survey 
respondent samples. Though the magnitude of impacts varies slightly between samples, the over- 
all pattern is generally the same. When there are multiple outcomes tested, the results are suscep- 
tible to false positives, so the research team performed a joint test to assess differences in multiple 
outcomes simultaneously. This test found that impacts on the two earnings and employment out- 
comes differ significantly between the full sample and the survey respondent sample (p-value = 
0.099). This result indicates that there may have been some differences between research groups 
on employment and earnings outcomes. 


Sensitivity Tests 


Two sensitivity tests further assess whether impact estimates are biased due to survey nonre- 
sponse. The first sensitivity test involves predicting the probability of survey response based on 
sample members’ baseline characteristics, then using these predicted probabilities to create 
weights that are then used in the impact model. The weighted results reflect the composition of 
the research sample, based on the sample’s observed baseline characteristics. 
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Appendix Table A.2 


Selected Baseline Characteristics of 30-Month 
Survey Respondents, by Research Group 


Program Control 


Characteristic Group Group 
Average age (years) 20.6 20.6 
Female (%) 53.1 52.8 


Race/ethnicity (%) 


Hispanic 34.4 36.8 

Black, non-Hispanic 60.2 58.3 

Other, non-Hispanic 1.5 1.3 
Ever employed (%) 71.8 72.6 
Worked in last three months (%) 28.4 29.7 
Has children (%) 19.4 21.1 
Has high school diploma or GED certificate (%) 61.5 64.1 
Serious barrier to employment* (%) 40.6 42.1 
Receives public assistance (%) 25.8 27.3 
Ever arrested (%) 23.1 25.7 
Sample size 1,272 759 


SOURCES: MDRC calculations based on data from MDRC's random assignment system and the YAIP 
management information system. 


NOTES: Statistical significance levels are indicated as follows: ***=1 percent; ** = 5 percent; * = 10 
percent. 

aThis measure includes ever having run away from home, been homeless, or lived in unstable 
housing; history of foster care; limited literacy or math skills; mental or physical disability; previous 
criminal conviction; and pregnancy or having a child. 


The second test, called multiple imputation, uses statistical modeling to predict survey 
responses for sample members who did not participate in the survey. Multiple predictions are 
generated to simulate the distribution of responses from which full sample estimates are produced. 
In other words, this analysis provides an estimate of survey-based impacts if the full research 
sample had participated in the survey. 
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Appendix Table A.3 


Selected 30-Month Impacts on Administrative Outcomes for the 
Research and Survey Respondent Samples 


90 Percent 
Program Control Difference Confidence 
Outcome Group Group _ (Impact) Interval 
Employed in the last year of follow-up (%) 
Research sample 78.2 78.6 -0.4 [-3.0, 2.2] 
Respondent sample, 30-month survey 80.8 79.1 1.6 [-1.2, 4.6] 
Total earnings in the last year of follow-up ($) 
Research sample 8,125 7,585 539 [-38, 1,117] 
Respondent sample, 30-month survey 8,327 7,585 742 * [74, 1,411] 
Enrolled in postsecondary institution (%) 
Research sample 22.7 24.8 -2.1 [-4.7, 0.4] 
Respondent sample, 30-month survey 24.5 25.1 -0.6 [-3.6, 2.4] 
Sample size 
Research sample (total = 2,678) 1,638 1,040 
Respondent sample, 30-month survey (total = 2,031) 1,272 759 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires and 
postsecondary education data from the National Student Clearinghouse. 


NOTES: Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Employment rates and earnings include both YAIP subsidized jobs and all other jobs covered by 

unemployment insurance. 


Appendix Table A.4 shows results from both sensitivity tests for two outcomes: employ- 
ment in the last year of follow-up and employment at the time of the 30-month survey.* Respond- 
ent sample rows show unweighted program impact estimates on employment. Weighted respond- 
ent sample rows apply response probability weights to the impact model. Research sample rows 
present program impact estimates for the full research sample as estimated via multiple imputa- 
tion. As shown in the table, impact estimates are virtually the same for both outcomes among 


Only two outcomes are shown in the table to conserve space, but both tests produced impacts similar to 
those of the respondent sample across all outcomes. 
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Appendix Table A.4 


Estimated Program Impacts on Selected Survey Outcomes for the 30-Month 
Survey Respondent Sample Compared with Two Sensitivity Tests 


Outcome (%) Program Impact 


Employed in the last year of follow-up 


Respondent sample 2.1 
Respondent sample, weighted to reflect full sample composition 2.1 
Research sample, with outcomes multiply imputed for nonrespondents 2.3 


Employed at time of 30-month survey 


Respondent sample 4.8 ** 

Respondent sample, weighted to reflect full sample composition 4.5 ** 

Research sample, with outcomes multiply imputed for nonrespondents 4.6 ** 
Research sample 2,678 
30-month survey sample 2,031 


SOURCES: MDRC calculations based on data from MDRC's random assignment system, the YAIP 
management information system, and responses to the Subsidized and Transitional Employment 
Demonstration 30-month survey. 


NOTES: Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

The weighted results weight survey responses so that they reflect the composition of the research sample. 
The multiple imputation results estimate program impacts on survey-based outcomes including probable 
responses for survey nonrespondents generated via multiple imputation. 


all samples, both in terms of size and statistical significance, suggesting little difference in pro- 
gram impacts between survey respondents and nonrespondents. 


Survey and National Directory of New Hires (NDNH) Employment 
Differences 


The results described above indicate that the difference between the self-reported current employ- 
ment and NDNH Quarter 10 employment impact estimates are only partially due to survey re- 
sponse bias. In order to investigate other sources of this difference, the research team compared 
various survey and administrative measures among the survey respondent sample, shown in Ap- 
pendix Table A.5. First, the team created a measure that captured whether survey respondents 
were working in the quarter of their survey interview, according to NDNH data.’ There was a 0.8 


3Not all survey respondents completed their survey in Quarter 10 after random assignment. 
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Appendix Table A.5 


Employment and Earnings Impacts Based on Administrative 
Records and 30-Month Survey Data Among Survey Respondents 


Program Control Difference 


Outcome (% Group Grou Impact 
Currently working, according to survey 65.0 60.5 4.5 ** 
Working in quarter of survey interview, according to administrative records 61.9 61.2 0.8 


Currently working, according to survey, or working in quarter of 


survey interview, according to administrative records 75.9 72.6 3.4 * 
Self-employed, according to survey 5.8 6.7 -0.9 
Sample size (total = 2,031) 1,272 759 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires and 
responses to the Subsidized and Transitional Employment Demonstration 30-month survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Results in this table are regression-adjusted, controlling for pre-random assignment characteristics. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


percentage point impact on this measure, which is not statistically significant and is much smaller 
than the survey-based current employment impact of 4.5 percentage points. Next, the team cre- 
ated a measure that captured whether survey respondents were currently working according to 
the survey or working in the quarter of the survey interview according to NDNH data. There was 
a Statistically significant 3.4 percentage point impact on this measure. Taken together, these re- 
sults indicate that the survey-based current employment impact is primarily driven by jobs that 
are not covered by the NDNH. The team also tested differences in survey-reported self-employ- 
ment and found a small, insignificant decrease in self-employment. Thus, the difference between 
self-reported and NDNH employment was not due to an increase in self-employment; rather, it 
appears that YAIP may have increased employment in the informal economy. 
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Appendix B 


Impact Variation 
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Chapter 3 presents an analysis of how impacts varied across subgroups, program locations, and program 
characteristics. This appendix provides more detail on the theories underlying these analyses, as well as the 
analytical approaches used in each analysis. 


Variation by Participant Characteristics 


The research team tested differences in program effects across four subgroups: educational status at base- 
line, length of disconnection from work and school at baseline, gender, and cohort of YAIP participation. 
The theoretical basis for these subgroups, as well as a summary of results, is described below. 


Educational Status at Baseline 


Jobseekers with high school equivalency credentials fare better in the job market, on average, than 
those without a high school credential. Further, most colleges and many vocational schools require appli- 
cants to have a high school equivalency credential in order to enroll in classes. To explore whether the 
YAIP model works differently for participants who would have access to more or fewer employment and 
educational options in the absence of YAIP, the research team tested whether impacts differed by educa- 
tional status at baseline. Appendix Table B.1 shows that, among those without a high school credential at 
baseline, YAIP increased high school credential receipt by 6 percentage points since random assignment, 
which, unsurprisingly, is significantly different from the null impact on those with a credential at baseline. 


Length of Disconnection from Work and School at Baseline 


As discussed in Chapter 1, disconnected youth are a heterogeneous group. For many, disconnect- 
edness is a temporary and inconstant state, but the longer one is disconnected from the worlds of work and 
school, the more difficult it is to reengage. The evaluation tested whether Y AIP works better for participants 
who had spent a longer (nine months or more) or shorter (less than nine months) amount of time away from 
work or school in the period immediately before study enrollment. As shown in Appendix Table B.2, there 
were no significant differences between groups in impacts by length of disconnection, suggesting that YAIP 
works equally well for more and less disconnected participants. 


Gender 


Nationally, disconnected young women are nearly four times as likely as connected young women 
to have children, and women are more likely to be the primary or sole caretakers of children in young 
families. Disconnected young men face their own challenges, including increased likelihood of living with 
a disability and lower educational attainment than both connected young men and disconnected young 
women.! These and other factors may limit one group’s ability to take advantage of YAIP or access em- 
ployment and educational opportunities relative to the other. As shown in Appendix Table B.3, female 
program group members earned $1,221 more in the last year of follow-up than their control group counter- 
parts, which is statistically significantly different from the impact among male sample members. Female 
program group members were also statistically significantly less likely to enroll in a two-year college than 


‘Burd-Sharps and Lewis (2017). 
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Appendix Table B.1 


30-Month Impacts on Primary Outcomes, by Educational Status at Baseline 


Has High School Diploma or GED Certificate 


Program Control Difference 


Outcome 


Currently employed or enrolled in school or 
training at time of 30-month survey (%) 86.1 


Education since random assignment (%) 


Earned high school diploma or 


equivalency certificate (based on survey)” 7.9 
Enrolled in a postsecondary institution 31.4 
Enrolled in four-year college 11.5 
Enrolled in two-year college 22.6 


Employment and earnings in last year of follow-up 


Employment (%) 82.6 
Average total earnings ($) 9,521 
Average number of quarters employed 2.6 
Average quarterly employment (%) 65.8 
Currently employed (based on survey) (%) 68.9 
Sample size (total = 2,666) 988 


Group Group 


84.3 


9.4 
33.3 
11.5 
23.4 


81.1 
8,788 
2.6 
64.0 
65.7 


656 


(Impact) 


1.8 


-1.5 
-1.9 

0.1 
-0.8 


1.4 

733 
0.1 
1.8 
3.1 


90 Percent 
Confidence 
Interval 


[-1.6, 5.2] 


[-4.1, 1.1] 
[-5.6, 1.8] 
[-2.5, 2.7] 
[-4.2, 2.6] 


[-1.7, 4.6] 
[-30, 1,496] 
[-0.1, 0.2] 
[-1.3, 4.9] 
[-1.3, 7.6] 


High School Diploma or GED Certificate 


Does Not Have 


Program Control Difference 


Group Group 


80.6 


46.3 
9.1 
2.7 
7.1 


71.9 
5,896 
2.0 
49.5 
58.2 


644 


76.5 


40.0 
10.7 
2.9 
8.4 


73.6 
5,636 
2.1 
51.4 
52.7 


378 


(Impact) 


4.1 


6.3 * 
-1.6 
-0.3 
-1.3 


-1.8 
260 
-0.1 
-1.9 

5.5 


Difference 
90 Percent Between 
Confidence Subgroup 


Interval _Impacts* 


[-1.0, 9.2] 


[0.3, 12.3] t 
[-4.7, 1.5] 
[-2.0, 1.5] 
[-4.1, 1.5] 


[-6.5, 3.0] 
[-623, 1,143] 
[-0.2, 0.1] 
[-6.0, 2.2] 
[-0.7, 11.6] 


(continued) 
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Appendix Table B.1 (continued) 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires, postsecondary education data from the National Student 
Clearinghouse, and responses to the Subsidized and Transitional Employment Demonstration 30-month survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 

Program impacts were calculated separately for each subgroup, using an ordinary least squares model and adjusting for pre-random assignment 
characteristics. Impact estimates were then examined for statistically significant differences across subgroups. 

Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a\WWhen comparing impacts between two subgroups, an H-statistic is used to assess whether the difference in impacts between the subgroups is statistically 
significant. Statistically significant differences between subgroups are indicated as follows: TTT = 1 percent; Tt = 5 percent; t = 10 percent. 

bThere are some discrepancies in reported educational status at baseline and reported high school credential receipt on the 30-month survey, resulting in 
reports of credential receipt during the study period among those who had a credential at baseline. This is likely due to a combination of recall issues and 
instances of people earning high school diplomas after already having a GED. 
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30-Month Impacts on Primary Outcomes, by Length of Time 
Disconnected from Work and School at Baseline 


Appendix Table B.2 


Nine Months or More 


Program Control Difference 


Outcome Group 


Currently employed or enrolled in school or 
training at time of 30-month survey (%) 81.1 


Education since random assignment (%) 


Earned high school diploma or 


equivalency certificate (based on survey) 20.9 
Enrolled in a postsecondary institution 18.7 
Enrolled in four-year college 7.9 
Enrolled in two-year college 12.6 


Employment and earnings in last year of follow-up 


Employment (%) 72.5 
Average total earnings ($) 6,933 
Average number of quarters employed 2.1 
Average quarterly employment (%) 52.5 
Currently employed (based on survey) (%) 59.7 
Sample size (total = 2,605) 788 


Group 


75.7 


19.5 
21.5 

6.9 
15.1 


72.9 
6,498 
2.1 
53.3 
53.6 


521 


(Impact) 


Bib. 


1.4 
-2.8 
1.1 
-2.5 


-0.4 
435 
0.0 
-0.8 
6.2 * 


90 Percent 
Confidence 
Interval 


[1.1, 9.8] 


[-2.5, 5.4] 
[-6.3, 0.7] 
[-1.4, 3.5] 
[-5.6, 0.6] 


[-4.5, 3.7] 
[-318, 1,188] 
[-0.2, 0.1] 
[-4.4, 2.8] 
(0.9, 11.5] 


Less Than Nine Months Difference 


Program Control Difference 


Group Group 


86.8 


22.9 
26.9 

8.8 
19.8 


83.8 
9,274 
2.6 
65.3 
70.0 


810 


86.7 


26.5 
26.9 

9.1 
19.7 


84.4 
8,604 
2.6 
65.3 
67.6 


486 


(Impact) 


0.1 


-3.6 
0.0 
-0.3 
0.1 


-0.7 
670 
0.0 
0.1 
2.5 


90 Percent Between 
Confidence Subgroup 
Interval Impacts* 


[-3.6, 3.9] 


[-7.8, 0.5] 
[-3.9, 3.9] 
[-2.9, 2.3] 
[-3.5, 3.7] 


[-4.1, 2.8] 
[-237, 1,577] 
[-0.1, 0.1] 
[-3.3, 3.5] 
[-2.6, 7.6] 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires, postsecondary education data from the National Student 


Clearinghouse, and responses to the Subsidized and Transitional Employment Demonstration 30-month survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Program impacts were calculated separately for each subgroup, using an ordinary least squares model and adjusting for pre-random assignment 
characteristics. Impact estimates were then examined for statistically significant differences across subgroups. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
a\W/hen comparing impacts between two subgroups, an H-statistic is used to assess whether the difference in impacts between the subgroups is statistically 
significant. Statistically significant differences between subgroups are indicated as follows: ttt = 1 percent; Tt = 5 percent; t = 10 percent. 


6S 


Appendix Table B.3 


30-Month Impacts on Primary Outcomes, by Gender 


Female 


Program Control Difference 


Outcome Group 


Currently employed or enrolled in school or 
training at time of 30-month survey (%) 84.4 


Education since random assignment (%) 


Earned high school diploma or 


equivalency certificate (based on survey) 20.3 
Enrolled in a postsecondary institution 24.3 
Enrolled in four-year college 10.2 
Enrolled in two-year college 15.8 


Employment and earnings in last year of follow-up 


Employment (%) 81.0 
Average total earnings ($) 8,420 
Average number of quarters employed 2.5 
Average quarterly employment (%) 62.4 
Currently employed (based on survey) (%) 65.8 
Sample size (total = 2,669) 833 


Group 


83.1 


19.4 
27.6 

9.2 
19.9 


81.0 
7,199 
2.4 
60.9 
61.3 


526 


(Impact) 


1.3 


0.9 
-3.4 
0.9 
4.1 * 


0.0 
1,221 ** 
0.1 
1.4 
4.6 


90 Percent 
Confidence 
Interval 


[-2.5, 5.1] 


[-2.8, 4.6] 
[-7.1, 0.4] 
[-1.7, 3.6] 
[-7.4, -0.7] 


[-3.5, 3.5] 
[439, 2,003] 
[-0.1, 0.2] 
[-2.0, 4.8] 
[-0.3, 9.5] 


Male 

90 Percent 
Program Control Difference Confidence 
Group Group (Impact) Interval 
84.3 78.8 5.5 ** [1.3, 9.7] 
24.6 24.2 0.3 [-3.9, 4.6] 
20.6 20.9 -0.4 [-3.9, 3.2] 
6.2 7.8 -1.6 [-3.9, 0.8] 
16.2 14.0 2.2 [-1.0, 5.5] 
75.2 76.0 -0.9 [-4.8, 3.1] 
7,801 7,999 -199 [-1,049, 652] 
2.2 2.3 -0.1 [-0.2, 0.1] 
55.8 57.5 -1.8 [-5.3, 1.8] 
65.2 59.0 6.2 * [0.9, 11.6] 

799 511 


Difference 
Between 
Subgroup 
Impacts* 


TH 


tt 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires, postsecondary education data from the National 


Student Clearinghouse, and responses to the Subsidized and Transitional Employment Demonstration 30-month survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Program impacts were calculated separately for each subgroup, using an ordinary least squares model and adjusting for pre-random assignment 

characteristics. Impact estimates were then examined for statistically significant differences across subgroups. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
aWhen comparing impacts between two subgroups, an H-statistic is used to assess whether the difference in impacts between the subgroups is statistically 

significant. Statistically significant differences between subgroups are indicated as follows: ttt = 1 percent; Tt = 5 percent; tT = 10 percent. 


female control group members, which is significantly different from the impact among male sample mem- 
bers. It is possible that the success female program group members experienced in the labor market dis- 
couraged them from enrolling in school. Interestingly, males experienced more employment benefits from 
the program in the short term, while females benefited slightly more in the long term. 


Cohorts 


As described in Chapter 1, the YAIP evaluation enrolled participants in three cohorts: July 2013, 
November 2013, and March 2014. Providers believed that each of these cohorts had unique characteristics, 
and engagement in employment and education can be cyclical. The implementation study found that there 
was high staff turnover among Y AIP staff, suggesting that implementation may have varied across cohorts. 
Thus, the evaluation tested differences in impacts across cohorts. As shown in Appendix Table B.4, July 
Cohort (Cohort 1) program group members experienced an 8 percentage point decrease in college enroll- 
ment relative to their control group counterparts, which was significantly different than the program’s ef- 
fects in the other cohorts. This result appears to primarily be the result of a decrease in two-year college 
enrollment, which is also a significantly different impact from the other cohorts. Enrollment in four-year 
colleges did not differ significantly by cohort. This result might be related to the timing of the July Cohort: 
Participants might have been so engaged in YAIP that they chose not to enroll in the fall 2013 school 
semester — the most common time of year to start college — potentially leaving them behind their control 
group counterparts in the college enrollment cycle. Additionally, November Cohort (Cohort 2) program 
group members were significantly more likely to have been employed or enrolled in school or training at 
the time of the 30-month survey than control group members, which is significantly different from the 
impact among sample members in the other two cohorts. 


Variation by Location 


As described in Chapter 3, this analysis tested variation in location-level impact estimates for both confirm- 
atory measures, as well as for earnings in the first year of follow-up in order to examine variation in impacts 
across locations during the program period. Using an approach developed by researchers Bloom, Raud- 
enbush, Weiss, and Porter, this analysis estimated adjusted empirical Bayes location-level program effects 
for each location, as well as the cross-location standard deviation of program effects.* The research team 
then computed a Q-statistic, which tests whether the estimated standard deviation of location-level impact 
estimates is statistically significant. Put another way, the Q-statistic tests whether program effects varied 
significantly across program locations. To produce the location-level estimates, this analysis uses a two- 
level hierarchical linear model with fixed location-specific intercepts and random location-specific program 
assignment effects. The model also controls for the individual baseline covariates used in the full sample 
impact model, as well as for an indicator of whether individuals enrolled in Y AIP after the beginning of the 
program. 


“Year 1 earnings impacts can serve as a measure of treatment contrast, capturing both the amount of subsidized em- 
ployment a participant worked and the relative quality of a participant’s job following YAIP, which would be a result of 
strong job placement services. 


3Bloom, Raudenbush, Weiss, and Porter (2017). 
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Appendix Table B.4 


30-Month Impacts on Primary Outcomes, by Cohort 


Outcome 


Cohort 1 
Currently employed or enrolled in school or 
training at time of 30-month survey (%) 


Education since random assignment (%) 


Earned high school diploma or 
equivalency certificate (based on survey) 
Enrolled in a postsecondary institution 
Enrolled in four-year college 
Enrolled in two-year college 


Employment and earnings in last year of follow-up 
Employment (%) 

Average total earnings ($) 

Average number of quarters employed 

Average quarterly employment (%) 


Currently employed (based on survey) (%) 
Sample size (total = 868) 


Cohort 2 
Currently employed or enrolled in school or 
training at time of 30-month survey (%) 


Education since random assignment (%) 


Earned high school diploma or 
equivalency certificate (based on survey) 
Enrolled in a postsecondary institution 
Enrolled in four-year college 
Enrolled in two-year college 


Employment and earnings in last year of follow-up 
Employment (%) 

Average total earnings ($) 

Average number of quarters employed 

Average quarterly employment (%) 


Currently employed (based on survey) (%) 


Sample size (total = 900) 


Program Control 


Group 


81.5 


25.9 
17.0 

6.5 
12.2 


77.4 
8,390 
2.4 
59.5 
63.5 


534 


86.4 


20.3 
24.5 

7.7 
19.0 


78.2 
7,613 
2.3 
58.1 
64.7 


551 
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Group 


81.0 


23.9 
24.7 

8.3 
17.1 


77.2 
7,979 
2.3 
58.6 
55.4 


334 


78.6 


23.6 
21.4 

7.6 
15.5 


75.7 
7,084 
2.2 
55.9 
62.4 


349 


Difference 
(Impact) 


0.4 


2.0 
-7.7 
-1.8 
-4.9 


0.2 
811 
0.0 
0.9 
8.1 


7.8 


-3.3 
3.1 
0.1 
3.6 


2.5 
529 
0.1 
2.2 
2.2 


KKK 


xk 


xk 


KKK 


90 Percent 
Confidence 
Interval 


[-4.7, 5.6] 


[-3.0, 7.0] 
[-12.2, -3.3] 
[-4.7, 1.1] 
[-8.9, -1.0] 


[-4.5, 4.9] 
[-228, 1,851] 
[-0.1, 0.2] 
[-3.5, 5.3] 
[1.7, 14.5] 


[3.0, 12.5] 


[-8.1, 1.5] 
[-1.4, 7.5] 
[-2.8, 3.1] 
[-0.5, 7.7] 


(22) 7.4) 
[-425, 1,482] 
[-0.1, 0.3] 
[-2.1, 6.5] 
[-3.9, 8.4] 


Difference 
Between 
Subgroup 
Impacts* 


tt 


tt 


tt 


tt 


(continued) 


Appendix Table B.4 (continued) 


Program Control 


Outcome Group 


Cohort 3 
Currently employed or enrolled in school or 
training at time of 30-month survey (%) 84.7 


Education since random assignment (%) 


Earned high school diploma or 


equivalency certificate (based on survey) 19.8 
Enrolled in a postsecondary institution 26.4 
Enrolled in four-year college 10.8 
Enrolled in two-year college 17.1 


Employment and earnings in last year of follow-up 


Employment (%) 78.9 
Average total earnings ($) 8,241 
Average number of quarters employed 2.4 
Average quarterly employment (%) 60.1 
Currently employed (based on survey) (%) 67.0 
Sample size (total = 910) 553 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New Hires, 


Group 


85.4 


17.4 
27.9 

8.7 
20.3 


83.2 
8,442 
2.6 
64.0 
63.8 


357 


Difference 
(Impact) 


-202 


-3.9 
3.2 


90 Percent 
Confidence 
Interval 


[-5.5, 4.1] 


[-2.5, 7.3] 
[-6.0, 3.1] 
[-1.2, 5.4] 
[-7.3, 1.0] 


[-8.7, 0.0] 
[-1,204, 801] 
[-0.3, 0.0] 
[-8.0, 0.3] 
[-3.1, 9.6] 


Difference 
Between 
Subgroup 
Impacts* 


tt 


tt 


postsecondary education data from the National Student Clearinghouse, and responses to the Subsidized and 


Transitional Employment Demonstration 30-month survey. 


NOTES: Rounding may cause slight discrepancies in sums and differences. 
Program impacts were calculated separately for each subgroup, using an ordinary least squares model and 
adjusting for pre-random assignment characteristics. Impact estimates were then examined for statistically 


significant differences across subgroups. 


Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 


a\WWhen comparing impacts between two subgroups, an H-statistic is used to assess whether the difference in 
impacts between the subgroups is statistically significant. Statistically significant differences between subgroups are 
indicated as follows: ttt = 1 percent; tt = 5 percent; T = 10 percent. 


This analysis found that program effects on confirmatory outcomes did not vary significantly across 
locations, but effects on Year 1 earnings varied significantly across locations. Appendix Figure B.1 shows 
location-level estimates of program effects on Year 1 earnings. The estimated cross-location standard de- 
viation of program effects on Year 1 earnings was $642 (p-value = 0.002). There was not enough variation 
in effects on earnings in the last year of follow-up or engagement in employment, education, or training at 
the end of the follow-up period for the model to calculate adjusted Empirical Bayes location-level effects. 
In other words, the best estimates of program effects at each location were equal to the estimates of effects 


across all locations, which are presented in Table 2.1. 
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Appendix Figure B.1 


Location-Level Adjusted Empirical Bayes Impact Estimates of 
Earnings in the First Year of Follow-up 


Impact Estimate and 95% 
Confidence Interval ($) 


Location 


SOURCES: MDRC calculations based on quarterly wage data from the National Directory of New 
Hires and YAIP management information system subsidized earnings records. 


NOTES: Sample size = 2,678. 

Each circle represents the adjusted empirical Bayes impact estimate for one of the 13 YAIP 
locations (location names masked), and the lines surrounding the circles represent the 95 percent 
confidence interval for each estimate. If the confidence interval crosses zero, the location-level 
impact estimate is not significant. Adjusted empirical Bayes impact estimates were obtained by 
using a two-level hierarchical linear model with fixed location-specific intercepts and random 
location-specific program assignment effects. The model also controls for individual baseline 
covariates. 


Variation by a Key Program Characteristic 


As described in Chapter 3, this analysis tested whether the presence of a job developer moderates program 
impacts. Although a number of factors might be associated with differences in impacts across programs, 
the analysis focuses on just one program characteristic due to statistical power. YAIP locations were typi- 
cally staffed with the equivalent of five full-time employees, but providers were left to determine how 
responsibilities would be distributed among program staff. Some locations had a designated unsubsidized 
job developer who spent the majority of time developing relationships with potential employers and refer- 
ring YAIP participants to jobs through these connections. Other locations did not have a job developer, so 
YAIP staff members would share job development responsibilities, generally squeezing this work in when- 
ever there was time. Providers believed that the latter arrangement led to a less-focused job development 
effort and that the presence of a designated job developer on staff was one of the location characteristics 
most strongly associated with improved employment outcomes for YAIP participants. Additionally, past 
implementation evaluations of transitional jobs have emphasized the importance of a strong focus on tran- 
sitioning participants to unsubsidized placements, and the presence of a job developer on staff serves as a 
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proxy for a general focus on this program component. The analysis found that the presence of a job devel- 
oper is not associated with impacts on either key outcome (results not shown in a table). 
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